Find common dependencies between your various builds and bake them into a base image. I think that’s about all you can do.

Do you want faster docker builds or smaller docker images? There are two different perspectives / optimization goals.