Hacker News new | ask | show | jobs
by Hydraulix989 1330 days ago
I struggle to find one correct remark in your comment. This take clearly suggests you have never worked on any similar infrastructure / distributed systems before at scale.

So you're telling me there's no backend or compute workers? And you're telling me that there are no opportunities for performance optimization? That they are already 100% native C++ services at full parallel utilization? That they've already tuned the size and cost of their ML models? That their model evaluation infra is fully saturated?

Storage is cheap. I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.

4 comments

> Storage is cheap

Seriously. They could back up all of Twitter on Backblaze for $7/month.

Hah, I chuckled bigly, nice one.
You should sit down and perform the storage calculation instead of struggling to find something correct in what I said because that will be a better use of your limited time.
> I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.

They already run their own datacenters (not uncommon for companies started in that era). They have moved partially into the cloud for data science stuff - GCP, not AWS, according to their engineering blog.

So… odds are that there isn’t nearly as much waste as everyone wants to believe.

DC is far more cost effective than public cloud for data intensive workloads.
Right, which is why they only executed a partial migration for certain workloads that make sense: https://blog.twitter.com/engineering/en_us/topics/infrastruc...

I think what really irks me is this meme that “obviously they have massive infrastructure waste.”

Does anyone have credible evidence that there is enormous waste in Twitter’s infrastructure? Just because Musk says “cut $1b in infra spending” doesn’t mean that there is $1b of things to cut, just lying about being wasted. Some of the smartest minds in our industry work/worked at Twitter. Are they that bad at their jobs?

What is actually going to happen is this: they’ll turn off all of their data warehousing stuff, blinding the business. They’ll cut their redundancies and backups, reducing the probability that they can do effective DR. They’ll reduce spending on observability so far that they won’t even know what’s going wrong (it’s surprisingly expensive). And that’ll get them to the $1b in cuts - by flying blind (both in a business sense, and a technical sense), and just hoping they don’t need to recover from a disaster.

> If so, even just moving to on-prem would save them the $1B easily

Except you can't wave a magic wand and shift the entire distributed infrastructure for one of the five most visited sites on earth on-prem. Twitter has two DCs. Running their infra on-prem somewhere would require reworking the entire company from the ground up in a way that would take years and cost substantially more than $1b.

Having their own DCs means they are on-prem already.