[djoldman]

Just FYI, the bigger companies all allow you to block crawlers via robots.txt:

    # Block Anthropic (Claude)
    User-agent: ClaudeBot
    Disallow: /
    User-agent: Claude-SearchBot
    Disallow: /
    User-agent: Claude-User
    Disallow: /

    # Block OpenAI (ChatGPT)
    User-agent: GPTBot
    Disallow: /
    User-agent: OAI-SearchBot
    Disallow: /

    # Block Perplexity
    User-agent: PerplexityBot
    Disallow: /

    # Block Google's AI Training
    User-agent: Google-Extended
    Disallow: /
    User-agent: Google-Extended-Factual
    Disallow: /

    # Block Microsoft's Search & AI Crawler
    User-agent: Bingbot
    Disallow: /

[Cthulhu_]

This only works for crawlers that play fair, unfortunately there's many that don't. Including ones that use consumer devices like TVs [0] and mobile phone apps that offer incentives to consumers (if they're even open about it) to use their internet connection to e.g. crawl websites. There's probably an army of hacked toothbrushes and the like doing the same thing.

[0] https://news.ycombinator.com/item?id=49000864

[spiderfarmer]

90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.

[6510]

I once had the dumb idea to make websites in pdf but the idea is growing on me. Perhaps it should even be in animated gif with <img> <map>'s

[frereubu]

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot...

I presume it would also cut you off even more from referral traffic.

[FinnKuhn]

They don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though.

https://techcrunch.com/2026/07/01/cloudflares-new-policy-pus...

[pixelesque]

Claude Bot still (at least last month, and it's been doing it for over a year now) seems to have a bug when traversing (at least my sites), wherein it drops the trailing slash of a directory (which is present in the a href tag), then makes the request to the subdirectory without the slash I put in the link, then Caddy automatically responds to that (via the built-in File handler) with a redirect telling it to add the trailing slash, and Claude Bot then makes the request again with the trailing slash that should have been there in the first place.

So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.

[globular-toast]

I won't eat your lunch if you put a sticker on your lunch box telling me not to.