Docker images in CI are typically just that: a tmpfs with a chroot and some network isolation. If you have it working once, youre pretty much guaranteed it will work again.
Doing this on bare metal with a script to clean the FS, ensure correct dependencies and maybe isolate the network (for safe unit tests), means you're just reimplementing much a non-trivial portion of docker or other container tools. Maybe that's worth it, but without justification, it just smells like risky NIH to me.