Hacker News new | ask | show | jobs
by kevinsundar 2527 days ago
I work at a FAANG and host level cpu is most definitely an alert we page on. Though a single host hitting 100% CPU isn't really a problem in and of itself (our SOP is just to replace the host), its an important sign to watch for other hosts becoming unhealthy. It might be overkill but hey theres mission critical stuff at hand.

For example: if you have a fleet of hosts handling jobs with retries, a bad job could end up being passed host to host killing each host / locking up each one as it gets passed along. And that could happen in minutes while replacing and deploying and bootstrapping a new host takes longer. So by the time your automated system detects, removes, and spins up a new host everything is on fire.
1 comments
learnfromstory 2527 days ago
Could you mention which FAANG so I can avoid applying or a job there? Large-scale software systems _must_ be designed to serve through local resource exhaustion. If you are paging on resource exhaustion of single host you are just paying the interest on your technical debt by drawing down your SREs' quality of life.

I stand by my beef with this article. The statement that "I've talked with engineers at Google [and concluded that a thing Google wouldn't tolerate is a must-have]" doesn't make sense. What I get from this article is you can talk with engineers at Google without learning anything.

link
kevinsundar 2527 days ago
Im not at liberty right now to name my employer but our systems are definitely designed to serve through local resource exhaustion. But we aren't talking about cheap hosts here. We generally run high compute optimized or high memory optimized hosts depending on the use case and if these generally powerful hosts hit 100% CPU or full memory utiliziation theres usually more going on than something random or simple so its important to have someone check it out.
link
packetslave 2527 days ago
A single host stuck at 100% CPU also has a nasty effect on your tail latency, in a system with wide fanout. If a request hits 100 backend systems, and 1 of them is slow, your 99th percentile latency is going to go in the toilet.
link
learnfromstory 2527 days ago
Which is a good reason to hedge and replicate but NOT a reason to alert on high CPU usage of single computers.
link
packetslave 2527 days ago
You definitely want to TRACK cpu usage on individual hosts, but, yeah, I would alert on service latency instead. Symptom, not cause.
link