|
|
|
|
|
by neurobro
2322 days ago
|
|
With a few exceptions like HD video hosting or sites doing a ton of computation for every request, the best response to point 1 is to just lighten the site, which reduces the load from bots and also improves the UX for humans. Rate limiting can also help in some circumstances. Or maybe consider... charging money. If you don't force botters to take extreme camouflage measures, bots are easily filtered out of logs (and offer a potentially useful metric of their own). If a business is threatened by "competitors" simply scraping published information, it's probably already doomed from the start. And I would posit that most site owners with this mindset originally got their data by scraping other sites, which is why they feel vulnerable. Compete over elements that are actually valuable! |
|