I can't comment on BuildKit, unfortunately, since I haven't used it. My experience comes from building bespoke systems (an image builder and a custom registry) fully from scratch, because we needed to have full control in order to achieve the performance we were aiming at.
Controlling both the builder and the registry is super nice btw, because they can work together. If the builder knows some of the layers already exist in the registry, it merely has to create and push the remaining ones (without downloading any of the other layers, not even those from the base image). That gives you near-instant builds once the biggest layers are cached in the registry!
Since builds usually happen in CI, and pulls happen elsewhere (e.g., a kubernetes node), in the end layers are only downloaded when the resulting container image is actually used.
Base image independent layers is something I’ve pondered about, but doesn’t feel compatible with things like apt. This conversation is giving me more reason to go implement lazy/fuse base layers though. My exports are already pretty fast due not not using tar+deduplication with similar layers, but pulling the base can still take several minutes.