Hacker News new | ask | show | jobs
by o11c 297 days ago
To be fair, a saner way to verify bots has been needed for a long time, and is not only relevant for AI bots.
2 comments

Yeah, the state of the art is reverse DNS and then checking that the forward DNS matches which is quite a mess and requires careful use of egress IPs and depends on the network for security. Actually signing requests is a huge improvement.

And while Cloudflare wants them to register which isn't great the standard does allow automatic discovery and verification of the signing keys which allows you to reliably get an associated domain which is very nice.

As the Cloudflare post indicates, most crawlers can be verified by IP address.