|
|
|
|
|
|
by nathants
846 days ago
|
|
|
there is very little reason to use services like big query. they are all insanely over priced. always use plain ec2 spot and s3. lots of smaller instances. fewer larger instances. single massive instance. whatever. fancy sql thingy, awk and grep, or whatever else. do your data processing with ephemeral spot priced compute and persist as little data as possible to s3. $2-5/hour gets an insane amount of ec2 spot. egress aside, no surprise bill is possible. empathy for op though. not a fun day. just a bump in the road though, keep on trucking! |
|
|
One place I worked at had a table with 100 billion rows. And some other tables as well. If a manager asked for an ad-hoc query, it was 5 minutes of writing a SQL query including JOINs (which didn't need to worry about which fields were indexed etc. e.g. you could write WHERE then a regex), and $15 and 5 minutes later I'd have the answer. Apparently 100s of VMs were started and stopped to answer that query, but it all happened automatically, at very low cost.