Sounds like it already is containerized, and that's the problem? A lot of places have CI setups that don't cache things anymore for various reasons. A while back there was a new YC startup announced whose product was essentially a SaaS for building Docker containers where they cache things for you.
Another cause of this problem is when CI workers are spun up and down on demand in the cloud, so VMs are constantly being set up from scratch.
The same reasons you can't just inline everything in a program, but break things into functions, modules, etc. Complex CI systems are, in my experiences, built up from components that are reused across multiple project builds and need to be encapsulated in a way that they can be plugged into any of those. Each component is containerized and caches as much as possible at each step, but that is still a lot of overhead.
Another cause of this problem is when CI workers are spun up and down on demand in the cloud, so VMs are constantly being set up from scratch.