[afpx]

Recently, I started helping out data engineering teams improve performance of their big data processing pipelines.

Man, was I shocked. Very smart, highly educated, mid-level and even senior software engineers seem to know very little about bits these days. When they'd run into a memory issue, their natural response was to just spin up a few more servers and throw another terabyte of memory at the problem. Makes sense, I guess - until their CFO saw their pretty exponential curve in infrastructure costs.

[hasenj]

It's lucrative for SaaS providers to convince developers that they don't need to care about performance. Just throw more memory at your problem. We'll be there to provide you all the hardware you need. wink wink.

[Illniyar]

And did knowledge of bits help solve the memory issue?

[MichaelGG]

I've done stuff where memory usage shrank from 30-60GB to under 2GB, and load times went from many minutes to essentially instant (mmap on the bits Vs needing to build objects).

[afpx]

That was implicit. Of course it did! :-) And, cut server costs in half, too.

[auggierose]

I like bits, too, but cutting it in half means it is still exponential ;-)

[afpx]

To be fair, the savings are way above 50%, honestly. But, I didn't want to claim a bigger number without digging in more, and 50% was a very conservative upper bound. But, yeah, you're right.