[anonytrary]

I remember when you could just stream in every tweet and reddit comment ever posted with a few lines of code. There used to be dedicated API methods for doing so. Now, there's all sorts of namespacing, rate limiting, pagination, and upper bounds on data access that make this impossible, or at least infeasible.

[sneak]

Twitter wants to capitalize on the fact that tweets are massively public, and they also want to capitalize on the fact that they are not.

Researchers are passing around lists of tweet IDs ("dehydrated", they call them) that can be "rehydrated" (that is, turned back into full tweets) if you have the right permission from twitter to do so.

The whole setup is really shameful.

It would be de-facto illegal to build a "Google for Twitter" today. I settled on doing it for ActivityPub/Mastodon because it's less likely I'll get sued into oblivion for creating a search engine that way.

[snisarenko]

Hi sneak,

I am working on some fediverse aggregator projects. Let me know if you want to talk.

https://mastodonia.club

https://pixelfed.club

[sneak]

You have no standard contact methods available on any of your sites that do not require registration. Consider publishing your email address; I am happy to chat. My email is in my profile and on my website.

[snisarenko]

Sent you an email

[banana_giraffe]

This reminds me of an endpoint that Google used to run. It was a never ending stream of RSS feeds. Anyone could subscribe (it was just a never ending HTTP GET, if I remember right), and be told which blogs had updated in near real time.

Nowadays you'd need to sign up for an API key, probably pay some amount of money, and provide twenty different forms of contact to use something like that. That's assuming it was allowed to exist in the first place, since it might take views away from google.com.