Hacker News new | ask | show | jobs
by wooly_bully 2240 days ago
In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.
3 comments

I live in a software engineering world professionally but my background is in traditional "neckbeard" Linux system administration. This ends up making me "DevOps" but honestly a lot of what I've ended up doing in my career is basic sysadmin for organizations that get a remarkably long ways before realizing they need it - things like telephony and video surveillance become really unreasonably expensive when you end up relying on a cloud service because you don't have the skillset to manage them in-house.

This is purely my opinion, but I think that 1) there is a strange shortage of IT professionals (people who are not software engineers but instead understand systems) in much of the industry today, and 2) a lot of tech companies, even those that are currently well functioning, might be able to save a lot of money if they hired someone with a conventional IT background. This is a little self-serving of course, but it really does astound me when I see the bills that some companies are paying cloud services to do something that is traditionally done in-house by an IT department. And not everything can readily be outsourced to some "aaS" provider, so on top of that you end up with things like software companies with multi-million budgets running an office network that consists of a consumer WiFi router someone picked up at Fry's - not realizing that they are losing a lot of time to dealing with how poorly that ends up working.

I think part of the problem rests in academia - at least in my area a lot of universities seem to have really backed off on IT programs in favor of CS. I went through an undergraduate program that involved project management, decision analysis, and finance courses because these were considered by the college (I would say accurately) critical skills for the IT field. But that program had an incredible two students and was widely considered inferior to the CS program with hundreds.

Another part of the problem though seems to rest in industry. The salary differential between "DevOps Engineer" and "IT Analyst" is incredible when in practice they end up doing mostly the same thing in a lot of small orgs. So I end up walking sort of an odd line of "I have a long background in IaC but I also know about conference room equipment." And I'm not saying that everything with a Cisco/Tandberg badge isn't overpriced, but Zoom rooms can end up costing just as much and seem to be less reliable - not surprising for a platform which, by practical necessity of the lack of IT support in many orgs, is built on the Silicon Valley time-tested architecture of "five apple consumer products taped together."

From my experience, large enterprises sabotage the effectiveness of internal IT with bureaucracy and politics in a misguided attempt to eliminate all possibility of mistakes being made.

It's usually done with the "let's pretend it is ITIL" process.

Let me give two examples where if I had been the client then I would absolutely have sprinted for the cloud if I could, or at the very least start talking it up as much better.

1) System outage, time to fix 5 hours and 3 minutes. The 5 hours was me sitting in front of my computer with screens open showing the problem and waiting for various managers/decision-makers to fly by and take a look as they were ping-ponging around the office panicking about what would be impacted by the fix. Everything that was going to get impacted was already impacted by the system not working, and I had to explain that to them multiple times. Towards the end of the day, I eventually got the go-ahead to do the 3 minutes of work to fix the system. This system being down had prevented another team from doing any work for the entire afternoon.

2) Two full days of politics and paperwork to get approval to do 30 minutes of work, all while the client was impatiently asking "is it done yet" every few hours.

This is accurate in my experience. Also keeping things up and being there in case shit hits the fan is a full-time job. You can't write features and also manage servers equally well. Unless you have no life I guess.
Totally. I've had personal colo'd servers for 20 years at this point. But I'm tired of knowing that at any point I might have to wake up and haul my ass down to San Jose to swear at some piece of failing gear. I'm excitedly moving it all into the cloud.
There is a middle ground. There are smaller companies where you can rent a bare metal server, usually with unmetered connection (priced by bandwidth). There is an on-call support 24x7 to replace any failing part when you call them. They can build custom servers and also give long term or bulk discounts.

I have a good experience with [1] (a smaller local company) and Hetzner [2], a bigger provider. Compare the prices with cloud. Especially if you need something RAM intensive.

[1]: https://www.superhosting.net/dedicated-servers

[2]: https://www.hetzner.com/dedicated-rootserver

How far are you from you collocation? Is moving that closer an option?
Colocation in SF proper is very spendy, which is how we ended up in San Jose. But waking up to go anywhere at 3 am to swear at gear is no longer on my list of fun activities. And there's no colo that will move the gear along with me when I go on vacation.
> there's no colo that will move the gear along with me when I go on vacation

This is actually the main one for me. I've managed our own servers for a decade with almost zero downtime, and very little time spent at the colo. But you cannot safely go out of town without having someone else around who is familiar enough with your setup to deal with an outage.

So moving stuff to AWS now almost entirely for that reason.

Learn it. Just like a new framework.

It's really not that hard.