Hacker News new | ask | show | jobs
by jaggirs 1083 days ago
Yeah no you cant just 'use selenium'. To keep the same scraping volume you might need thousands of accounts and 10x the compute.
1 comments

It’s not a little “use selenium” switch you can click, but it absolutely is an option (and there are others) if the barrier is simply to have an authenticated account and be logged in.

If these data scraping operations are as sophisticated and determined as he claims this measure is insufficient and actually it really hurts Twitter far more than it helps. Case in point: we stopped sharing Twitter links because when you click them in most iOS apps it opens up an unauthenticated web view and presents you with a login screen. So we just collectively decided “ah ok no sharing Twitter” and moved on.

I’m sure there are companies scraping Twitter. I just don’t buy that it’s as big of an issue as he claims it is, and that preventing people from viewing tweets without logging in is a way to mitigate against that (I’d first look at banning problematic IP addresses first, personally).

To me it’s either:

1) a very poor and very temporary mitigation against scraping, that could be bypassed with a bit of effort

2) an experiment in optimising metrics - Musk sees lots of unauthenticated users consuming Twitter, tries to steer them into signing up

3) it’s all just a big mistake

Option #2 makes the most sense to me, but frankly none of them are good