You may not be able to stop AIs from crawling web sites through technological means. But you can confiscate all the resources of the company that owns the AI.