Hacker News new | ask | show | jobs
by _slih 121 days ago
the facebook bug is funny but the real issue is that robots.txt is a gentleman's agreement from 1994 and half the crawlers in 2026 don't care what it says.