Hacker News new | ask | show | jobs
by godrae369 107 days ago
Hey! Saw your post about GPTBot eating up your Cloudflare Workers quota. The brutal truth is that relying on robots.txt in 2026 is like putting a 'please do not enter' sticky note on a bank vault. AI scrapers are notoriously ignoring them or rotating IPs. You are literally paying Cloudflare so OpenAI can train their models. I build custom infrastructure defenses for SaaS founders. Instead of hoping they respect your robots.txt, I can set up specific Cloudflare WAF (Web Application Firewall) rules and Edge Workers that fingerprint AI scrapers (even when they spoof user-agents) and drop the connection at the edge before it hits your billing quota. If you want to permanently lock them out and protect your server bill, let's chat. I can share the JSON ruleset for your Cloudflare dashboard.