[hereonout2]

I've done both, tiny scratch based images with a single go binary to full fat ubuntu based things.

What is killing me at the moment is deploying Docker based AI applications.

The CUDA base images come in at several GB to start with, then typically a whole host of python dependencies will be added with things like pytorch adding almost a GB of binaries.

Typically the application code is tiny as it's usually just python, but then you have the ML model itself. These can be many GB too, so you need to decide whether to add it to the image or mount it as a volume, regardless it needs to make it's way onto the deployment target.

I'm currently delivering double digit GB docker images to different parts of my organisation which raises eyebrows. I'm not sure a way around it though, it's less a docker problem and more an AI / CUDA issue.

Docker fits current workflows but I can't help feeling having custom VM images for this type of thing would be more efficient.

[kevmo314]

PyTorch essentially landed on the same bundling CUDA solution, so you're at least in good company.

[hereonout2]

Yep, then I have some projects that have pytorch dependencies which use it's own bundled CUDA and non-pytorch dependencies that use a CUDA in the usual system wide include path.

So CUDA gets packaged up in the container twice unless I start building everything from source or messing about with RPATHs!