Hacker News new | ask | show | jobs
by dabeeeenster 356 days ago
I mean, that's just not true.
1 comments
esskay 356 days ago
Which part? It's widely established and known that many AI crawlers are ignoring the robots.txt file, perplexity being one of them [1]

[1]https://www.tomshardware.com/tech-industry/artificial-intell...

link