[nonameiguess]

It coincides with the rise of web-based centralized services. If you're creating Windows, the classic Office suite, classic Photoshop, MATLAB, whatever software was popular 25 years ago that users installed and ran themselves, your part of the software lifecycle effectively ends when you ship, minus some minimal level of customer support and maybe training. Feature development, bug fixes, and hopefully testing is the vast bulk of what you do as a software company.

Fast forward to today where everything is running on servers and you're now responsible for installing, updating, monitoring, and scaling yourself, and probably also want some level of automation in the test and release processes. The other half of the software lifecycle is now core to your own business. The engineering challenges posed and the skillsets to handle them overlap with product development, but are not identical. So specialized roles have evolved to fill the gap.

As for the apparent demand, this is the part of the software lifecycle that CS and SE degree programs largely gloss over or don't teach at all, and there are no bootcamps for it. It's also very difficult to self-teach for the same reason there aren't a whole lot of self-taught high-energy physicists. It's easier to build out a homelab than a particle collider, but largely you need equipment beyond a personal laptop. This means most of the people doing this kind of work learned on the job at some point, and that means they're more senior and rarer.