[rosser]

CPU time is so ridiculously cheap compared to disk IO — even on SSD — that it's pretty much always a win, sometimes massively so.

My comment during a previous instance of this discussion:

https://news.ycombinator.com/item?id=5592341

[mixologic]

I think you're comparing cheap in terms of time, Im talking about in terms of resource utilization and saturation. I think it would be clearer to show what the exact tradeoff is in terms of cost. In an arbitrary scenario where you have 100$ worth of ssd, and 100$ worth of CPU, and this strategy saves 75% of the SSD space at the cost of 1% more utilization of CPU, then yeah, its a total win. But isnt your example showing that it takes 5 times as much cpu time compressed vs uncompressed?

[rosser]

Just using my own example, it's roughly a 5x increase in CPU time for a > 10x reduction in disk space (and commensurate reduction in disk IO — including writes, which is an important consideration when you're using SSD). Let's just say 10x though, for sake of easier math.

Assume $1/gbyte storage costs. We're not talking about consumer-grade drives here, right? It's probably worse than that anyway, because you're using some form of RAID, too. Aren't you?

Assume further a $1000 8-core CPU, as that's what the machine in my example uses (it's actually slightly more expensive, but again: easier math), and a 3 year depreciation schedule. That's roughly 1000 days, or $1/day, $0.125/core-day, or $0.000087/core-second.

Storing that log file uncompressed costs you $8.817 in storage, and $0.00085 in CPU time, if the process uses 100% of a core for the duration. Storing it compressed costs you $0.828 in disk space and $0.0038 in CPU time — again, if you're burning an entire core for all 44s.

I think $0.8318 is less than $8.81785, but feel free to check my math...

EDIT: Yes, I know I'm conflating the ongoing cost of storage and the incidental cost of CPU time. I'm also ignoring the cost of power and cooling, leaving entirely aside the difference between storage milliseconds and CPU nanoseconds, and, and, and. I guess, if it's not obvious that compression is pretty much unequivocally a win with modern CPUs, then I don't know what else to say.

[aquadrop]

That math stands only if your server has spare CPU utilization.

[rosser]

I disagree. Being IO starved is far more likely, and has a vastly greater impact on the utility and capability of a host, than being CPU starved.

[aquadrop]

Sure IO starving is more likely, but on analytic DB (which is supposed use of showed tech) you probably will do some complex queries with joins, filtering, aggregation etc. That can load CPU pretty well. So you always need to check where the bottleneck is.