[azeemba]

Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.

[mx7zysuj4xew]

As it should be. Bots are still user agents much like browsers

[wmf]

If you're paying per token for AI, you can also pay a smaller amount to use the Web.