Hacker News new | ask | show | jobs
by albertgoeswoof 858 days ago
Today I asked a devops engineer to tell me how much time a long (3 seconds avg) api call was spending on database queries, application logic, and network etc. He couldn’t understand the request and instead opened up the azure console and recommended we increase the number cpu cores / memory if performance is an issue.

I look at posts like this and cry.

4 comments

I keep telling people that a hundred buses will let you take a hundred times more people, but nobody will get to their destination a hundred times faster.

Inevitably this comment is followed by quiet blinking as they digest this and then this question: “Are you saying we need to scale up one hundred times bigger?”

Sigh…

Really makes one wonder what kind of thought process these people go through, and what kind of education they actually had.
On two occasions I have been asked, — “Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?” In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

— Charles Babbage, Passages from the Life of a Philosopher (1864), chapter 5, Difference Engine No. 1

Your DevOps engineer needs to learn more about observability. Jaeger or similar could be helpful.

Also, with DevOps pushing out traditional administrators, companies are often spending way more on infra than needed.

Implementing traces should be done by devs, not the infrastructure team. Devops should implement/support the platform that supports traces.
In Azure, use Application Insights for this. It's easy to set up and shows distributed performance traces across multiple services in a GUI.
Disagree, DevOps teams should be looking for resources that are being hit hard unnecessarily and request moves to better solutions when possible.

DevOps teams should be looking at CPU spikes, and should be performing RCAs, and they should be maintaining resources in a healthy state, and they should reject/revert changes and notify problem areas in code by product focused devs.

Product devs, for the most part, are only implementing human lex traces to debug business logic when it arises. Product devs are not equipped with the knowledge to identify system errors that are not "bugs in the code", i.e. they will not be good at telling you why SPROC_LXC1 fails as a result of making a ExcelParserFactoryFactory

> Today I asked a devops engineer (..) He couldn’t understand the request

"Engineer"?!? Given above description, that makes me cry.