[boredemployee]

At the beginning of my career, I incurred some hundreds of dollars in losses with BigQuery and Google Cloud Function. The problem with these services is that they are easy and intuitive enough for a beginner to use, but a nightmare to maintain.

[winrid]

That's nothing. Wait until you incur $150k of Lambda costs in a day!

[boredemployee]

Did it happen to you? sorry to hear!

[winrid]

Project ended up saving 10x that per year, so wasn't a huge deal. Quickly rewrote it to run as a traditional server, though.

[ceejayoz]

Invocation loop?

[winrid]

nah I was using it for benchmarking downstream services and the benchmark "worked" in that it overloaded the downstream services and accidentally had the lambas waiting too long for responses (we had to wait, to simulate real load and connections).

It was originally estimated like 10k or something per test which was approved at the time (had like 3 level of management all down my neck for getting it out, hence using lambda originally).

We did deliver, just needed one more sprint to rewrite it as a distributed system on servers. ;) Moved to like 20 machines w/ 128gb of ram that we could spin up as needed (testing millions of events a second, system in NodeJS!)

[hilti]

I‘ve had almost the same experience. First I was super impressed how easy it is to get data into BigQuery and retrieve it using their IDE.

But really soon I noticed the slow startup … simple queries took too long (1.2 sec vs milliseconds in a traditional database)

Then I learned a lot about BigQuery views. That helped a little.

At some point I simply wanted to export data. New Google tools needed to be learned: Cloud Storage, Data Flow.

After 18 months of using BigQuery on roughly 850 million rows, I switched back to a traditional database.

[dharmab]

I'm glad you learned that lesson for less than $1k. I think everyone who's ever worked with large amounts of data in BigQuery has a story like that, and sometimes the number is six or seven digits.

[fifilura]

Most of the time, it feels a little bit embarrassing, but the cost is just a very small part of your regular salary and overall operating cost. If your boss hits you with this, they don't have the correct perspective and priorities.

My record is $20k and it raised some eyebrows. But it was not really a mistake, just a sub-optimal backfill.

The data was filling a need for making appropriate business decisions, and compared to all the money lost by business developers making investments on a hunch, this was a very small bump in the road.