Hacker News new | ask | show | jobs
by zupa-hu 460 days ago
Every automated crawler follows human-driven actions.
2 comments

Conversely, every browser is a program that automatically executes HTTP requests.
Yet they respect a lot of things meant for machine to machine interaction. Like server return codes, cookie negotiations, and CAPTCHAs if they behave a certain way.

So they sometimes hit bollards and turnstiles made for other types of code which executes HTTP requests. So they're bots basically, but better (or suitably) behaving ones.

Browsers let you visit websites without regard for robots.txt.
Welcome to "Context".