[jjoe]

I'm not familiar with GAE and I'm suspecting this won't work there. But we've recently helped a client "deal" with enormous bot traffic. I dubbed the whole thing the "bot-split approach". One LB routes based on user agent. One traffic line goes to a heavily-cached server running Varnish and is hooked to a "stale" DB (updated hourly). And one traffic line dedicated to realtime/user traffic with almost zero caching. Heavy caching keeps the bot box alive with an OK load while the real user node has plenty of wiggle room.