Archival Efforts

[Chud]

In the last forum, I considered writing a script that could archive hundreds of Twitter accounts simultaneously. After the backlog has been retrieved, such a script would only have to run in the background for a couple minutes a day to keep the archives up-to-date. All tweets, retweets, attachments, profile pictures, and contextual replies would be saved to disk. Archives would have a web interface; you could either browse them offline on your own computer, or host them on a server for public access.

I have also considered implementing a system of "continuity" for jannied accounts. The owner of an archive would be able to create a "meta-handle" combining multiple living and dead Twitter handles (e.g "&EarthRabbit" => ["earth_inaba", "TerrRabbit", "TerraRabbit1488"]). The web interface to this handle would merge the accounts' feeds. It would also serve twitter card metadata (similar to that used to format news articles), so that a link to an archived jannied tweet would look like an ordinary quote tweet. This might get the host in legal trouble, so it's only an afterthought.

I have since started working on such a tool. Updates will be poasted in this thread.

[kevin]

That’s a good idea. I’ve looked into the Twitter API. Keep me posted

[BillyONare]

This is a hugely important project. Even Moldbug was talking about something exactly like this. Twitter will make your life hell. Godspeed.

[Chud]

I have admittedly been slacking on this project... mostly in the process of conceptualizing how such a tool would maintain continuity between individual "scrapes". It is a hugely important aspect of the project, so I must put some more thought into it.

[Guest]

@Chud , consider this: https://github.com/hartator/wayback-machine-downloader

it could be useful

[Suomi]

Is twitter search function really the best way of going through someone's old tweets?.. because it's really terrible, shows only few tweets unless you go day by day

[Chud]

Is twitter search function really the best way of going through someone's old tweets?.. because it's really terrible, shows only few tweets unless you go day by day

Nitter search function is much stronger.

[Guest]

This has already been done by some antifa guy funded by the German government - see his github for an extensive suite of tools to track account name changes, suspended accounts as deleted tweets as well as his archive of over 25 million tweets!

https://github.com/travisbrown/cancel-culture

https://github.com/travisbrown/deleted-tweets

Personally I hate the idea of allowing antifa jannies unlimited means to dox everyone who has ever posted anything remotely right-wing but c'est la vie.

[Leverkühn]

Those in the know can easily spot who is who, even people who don't make themselves particularly recognizable. Outsiders can't and that's great.

"I see this account has retweeted 20 lolis in a row and also used the nigger babble translator gif. Must be one of our guys"

[Guest]

Nitter seems like a good base since it already has many annoying parts (like refreshing tokens just to see tweets) done. It has archiving on the roadmap but not implemented yet.

If you think an account is going to be jannied, you could submit some of their posts to archive today. https://archive.ph/2VaOB
I saved a bunch of NFG posts there https://archive.ph/https://twitter.com/groyper_nick/*

I prefer archive today to archive org because
1) archive org tries to save pages "as they are", which means it tries to load all the garbage javascript bloat on twitter.
archive today turns it into nice flat quick loading html, I think it also clicks the 'show sensitive content' button.
2) the (presumably slavic) guy running archive today doesn't respond to dmca removal requests.

One problem is twitter's always increasing censorship is making it difficult to even read many threads and accounts, e.g. https://i.ibb.co/gmgcMdr/restricted-post.png

[Chud]

One problem is twitter's always increasing censorship is making it difficult to even read many threads and accounts, e.g. https://i.ibb.co/gmgcMdr/restricted-post.png

I spoilered that spider cunt as a joke and it probably made my whole thread less accessible