[wincy]

So what does one of these full time data center jobs look like, day to day? If I’m a software engineer I feel like I’d have to move and get a pay decrease to actually work at one of these? I mean until AI finally puts me out of a job. I guess I wouldn’t really be qualified to work one of these jobs?

[protocolture]

The blokes I meet who work 8x5 from a data center tend to spend their days installing hardware, deinstalling hardware, providing remote hands (usually cabling, sometimes console access)building racks, managing power supplies and maintaining asset registers. And escorting idiots like me around when things get technical.

[cyberax]

It's not easy, actually. You will likely need to be a licensed electrician or a licensed plumber. Both occupations require around 4000 hours of apprenticeship.

Some states don't need a license for low-voltage work, so you might be able to do data wiring.

[RijilV]

If you've never had an opportunity to spend time in a datacenter as a software developer, that's unfortunate but also far too common. What things look like on the inside vary company to company. Generally you're in an OSHA-abiding environment, so safety shoes, ear and eye protection, sometimes gloves.

There's a variety of roles. Security, electricians, HVAC engineers, generally some type of site foreman-ask role, logistics (depending on the size of the place), and technicians (for a lack of a better word, feels like every place calls them something different). There's a variety of roles that often float between sites or oversee many sites, depending again on the scale of the place. AWS is huge. Bigger than you're imagining, so there's quite a few levels deep and include real estate folks as well as construction roles. If you go and look at job postings, you'll even see roles for nuclear engineers at some companies.

But generally what you're talking about here are what I'm calling the technicians. They're responsible for wheeling racks into place (depending on the company they may also be responsible for unloading the trucks). Cabling is nearly always outsourced these days (though not the design of the cables), so rolling a rack into place generally involves securing it to the floor and connecting power, data, and more often than not now-a-days liquid cooling.

The other part of their job is "troubleshooting" failed hardware. Again, really depends on the company. Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up. I'm sure there are some great places to be a tech where you get to rip apart servers and bust out the multimeter, that to my understanding is not how the hyperscalers who actually hyper-scale do it.

There's some cleaning, parts management, destroying broken hard drives, shoveling snow off the roof (no lie), and a variety of other odd tasks.

If you ever have the opportunity to check out one of those places it can be a riot and a real eye opener. Depends again on the company though, some of those places have insane security (metal detectors, badge+pin, turnstile door procedures) which make visits super un-fun if they're even allowed outside of legit business reasons. Other companies... well I'm glad that's not where I store my data.

Back "in the day" (2005 give or take a handful of years) techs would often write their own automation and even build some simple services.

And yes, the jobs don't pay particularly well depending upon what it is. Electricians and such command decent wages, but the security guards and techs don't make crazy amounts. I think folks doing contract cabling can come out ahead.

Anyhow, SWEs are wildly insulated from the realities of what things look like on the ground. Maybe that's a good thing, IDK.

[martinpw]

> Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up.

Very true. I've heard stories of how technicians struggle with friends/family perceptions here. Since a lot of these datacenters are in rural communities, they are perceived as being technical wizards to be working there. But in reality they are doing as you say - just following a preprogrammed script with very little scope for any sort of creative problem solving.

[nunez]

Not the poster, but I highly recommend checking out your company's data center if you can (and if your company has one). Nothing like seeing where your apps or data actually land (and the physical security put in place to protect them)

You'll especially be in luck if your company is an old and has a mainframe or two. Those are incredible to behold. Masterful engineering.

[Henchman21]

You're likely unqualified. You'll be competing against a pool of people who are qualified and have worked these jobs.

I used to do SWE. Early in my career I realized that DCO/DCE was not only an entire discipline unto itself, but that the rest of the "team" looked down on it because it required physical labor. Perfect, I don't feel like I've done Real Work unless I do something in the physical world. Hence my DCE jobs. I would not be able to do SWE today.

But I'll tell you this: way more than once I've had a cocky new hire from the HFT I work at wither under simple tasks: Please re-balance the load across all phases of these PDUs. Please groom & remove dead cables from wherever. But you don't have time to figure it out on the fly. You have an hour to have it DONE.

Anyway, I think your intuition serves you well. :)

[fc417fc802]

Why would a datacenter employ an on site software engineer though? Anyway amazon already has plenty of those in house.

Outside of construction I don't believe datacenters employ many people locally.

[JumpCrisscross]

> Why would a datacenter employ an on site software engineer though?

It would be rather silly is a multi-billion dollar investment went down because, for some reason, admins couldn't remote in.

[jubilanti]

That's startup to mid sized traditional company thinking. Not at the hyperscaler enterprise scale.

Anybody working in even classic datacenter physical ops already knows how to plug a KVM with a cell modem into a box to let the engineers remote in. That's assuming the racks aren't already built to support this natively these days.

Come on, this is the industry that is going gangbusters on the fetish of mass unemployment and deskilling, you don't think they're doing everything they can to have to only hire a few local bodies at minimum wage to basically pull a bad rack out and slot a new spare in?

[mlrtime]

To build one or work in one after it's built?

There are little to no SWEs in a large AWS datacenter.