Hacker News new | ask | show | jobs
by imtringued 2016 days ago
It highly likely that this person caused a $72,000 bill on all the websites he has crawled. It's just that the cost is spread out over multiple websites so it is not noticeable.
1 comments

I don't think so, serving a page from cache is far cheaper than requesting, crawling, and storing that page in a database. Cloud comes with a premium, too.
While that’s true, not everything can be cached, and many websites run expensive code to assign a session to each new “user.” Larger sites generally learn to avoid that or have the infrastructure to accommodate it, but even moderate-sized blogs and forums probably can’t cope with that scenario all too well.