|
|
|
|
|
by plaintosapp
39 days ago
|
|
Yes exactly — it matches against a database of 18
known AI bot user agent tokens (GPTBot, ClaudeBot,
CCBot, Bytespider etc.) plus their known IP ranges
where available. GPTBot for example publishes its
IP ranges officially so we can match on both UA
and IP. The spoofing problem is the hard one. Bots that
fully spoof Chrome headers are invisible to any
UA-based tool including this one. The honest answer
is that BotCost catches the "polite" bots that
identify themselves — which covers the major AI
companies (OpenAI, Anthropic, Google, Meta) since
they all self-identify. The truly malicious scrapers
that spoof identities are a harder problem requiring
behavioral analysis. So it's accurate for what it is — catching known
AI training and search crawlers — but not a complete
bot detection solution. |
|