Hacker News new | ask | show | jobs
by moebrowne 103 days ago
> We constantly see crawlers trying to do cache busting

Do you have a source for this? Not saying you're wrong, I'd just like to know more

1 comments

Not really, given that the work we do in that direction isn't exactly public. You can recreate the scenario though. Spin up a wiki of some sort, scrapers love wikis, ideally enable some form of caching, and just sit back and watch scrapers throw random shit in the URL parameters.