Hacker News new | ask | show | jobs
by cr4zy 5281 days ago
I'm pretty sure this uses the map reduce API which has a lot of overhead in the datastore. In principle map reduce is nice because it could make very large jobs fast. But since Google engineers don't pay for anything, they optimized for time, not cost.

And with regards to your script, you can't just delete 3k keys in one request. If you want I'll send you the script I've adapted for jobs that make large changes to the datastore.

2 comments

From my experience purging data via MapReduce API use a lot less write quota than admin interface (but with a bit of instance hour overhead which doesn't seems like a problem)

I can't remember the exact number but it was about 10 times less than deleting via admin interface and finish in 5 minutes rather than 3 hours.

I meant that I needed 3k requests to finish the job, deleting 1k entities in each request :).