Y
Hacker News
new
|
ask
|
show
|
jobs
by
_slih
121 days ago
the facebook bug is funny but the real issue is that robots.txt is a gentleman's agreement from 1994 and half the crawlers in 2026 don't care what it says.