|
|
|
|
|
|
by toofy
139 days ago
|
|
|
it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something? implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.” |
|
|