|
|
|
|
|
|
by bloppe
1258 days ago
|
|
|
robots.txt has no force of law. Google would just be embarrassed to be caught disregarding it. Less reputation-focused crawlers don't care. With AI, it would be much harder to catch them red-handed since it typically transforms it's inputs enough to be hard to trace. The risk to reputation would be zero, so who would care? |
|
|