Hacker News new | ask | show | jobs
by pbhjpbhj 1234 days ago
Google were processing JavaScript for crawling websites more than a decade ago (a quick search suggests since ~2008).

It's not GPT that's having the issue it's what's feeding GPT the website.

1 comments

Is there any literature out there on how they do it at large scale?

From my understanding it’s tough and expensive ( selenium and rotating resedential ips) am I misinformed?

Google doesn't need residential IPs, since websites tend to treat Googlebot specially
Google purposefully obfuscate the details, AFAICT. I've not done SEO for ~5 years, I'm not sure I know anything useful on the subject any more.