[shortrounddev2]

The longer my career goes on, the less a lot of these cloud services make sense to me. I know why we have AWS deploy our production instances, but why are all of our staging and dev instances running on EC2? Can't we just buy an old PC and put it in the office and run dev builds on that? Sure it might be slow, but it's not meant to handle 100k QPS, it's just for QA

[heisenbit]

Sorry, this won't cut it anymore. You would loose all the fun instabilities when the cluster is upgraded again, the round trip times could not be optimized by distributing things further, the magic of all the different translation layers between you and the hardware would vanish. Worst of all is not all the money and headache you would save but you also would save time. Why would you do that?

[BossingAround]

Right, much better to deal with all of those issues in production only.

[tguvot]

in higher management there is aversion to any spend on physical stuff. where i work, we as everywhere look to cut costs. one of ways allowed to save ~$1.5M on aws costs over 5 years, by one time investment of $150k into physical equipment. It fell through.

The other aspect it's that somebody needs to manage old PC. usually there needs to be a team managing this infra and it creates friction between developers and infra team. so devs run into cloud and try to manage everything on cloud themselves. but at some point arrives consolidation and new team dedicated to managing same infra in aws.

[alephnerd]

> Can't we just buy an old PC and put it in the office and run dev builds on that

It's cheaper to pay for those cloud resources than hiring 2-3 more people.

[shortrounddev2]

I think if you had a whole server rack, you might have to hire someone for that, but if you're just running one or two servers with containers to run your dev builds, you could get that set up on K8s and manage your deployments the way you would with AWS. I don't think it would require hiring new people, an existing devops guy should be able to do that

[alephnerd]

It's still additional operational overhead for marginal RoI.

If you're an organization that already has a SRE or Infra Eng hired, there's no point supporting a niche setup when every other team is using a prebuilt and troubleshooted environment.

Edit: Can't reply to OP atm, but he makes a good point. For a higher margins and less commodified segment like ML Compute, on-prem computing still makes financial sense.

That said, that's still managed by actual SWEs not a generic Sysadmin, due to intricacies of GPU Architecture and Model Development, and OP is extremely underskilled for this role.

[shortrounddev2]

I think if all you're running is web microservices, then it's probably not worth it. But my company builds machine learning models offline and I feel that if we bought a dedicated GPU machine (literally just build one and put it in the office, not a rack mounted thing), we would save money in the long run after building a custom training pipeline

[mlinhares]

Why would you have your QA environment unlike you prod environment? Even the infra setup should be as close as possible, so you can test infra changes in QA before promoting them to prod.

I can't see how it would make sense to deploy in two completely different environments that are not alike at all.

[alephnerd]

> I can't see how it would make sense to deploy in two completely different environments that are not alike at all.

Because you are not saving that much money by having multiple bespoke on-prem environments per team.

Troubleshooting, management, and security headaches grow.

It's just easier to have everyone use a pre-built and troubleshooted environment.