[alephnerd]

It's still additional operational overhead for marginal RoI.

If you're an organization that already has a SRE or Infra Eng hired, there's no point supporting a niche setup when every other team is using a prebuilt and troubleshooted environment.

Edit: Can't reply to OP atm, but he makes a good point. For a higher margins and less commodified segment like ML Compute, on-prem computing still makes financial sense.

That said, that's still managed by actual SWEs not a generic Sysadmin, due to intricacies of GPU Architecture and Model Development, and OP is extremely underskilled for this role.

[shortrounddev2]

I think if all you're running is web microservices, then it's probably not worth it. But my company builds machine learning models offline and I feel that if we bought a dedicated GPU machine (literally just build one and put it in the office, not a rack mounted thing), we would save money in the long run after building a custom training pipeline