I exchanged an email with the Product Manager of some Azure DevOps component that comprises the build pipelines. They’re well aware of the need for caching. Hopefully it comes soon!
Unfortunately, they've been aware of this problem for years - I'm an Azure DevOps fan, but I really just don't understand why they didn't fix this an age ago :/