[bigbuppo]

Robots are supposed to behave. It was a solved problem 30 years ago until AI bros unsolved it. Any entity that does not obey robots.txt is by definition a malicious actor.