There are a number of things that I reach for in such situations, and some combination of Nix/Flox/Bazel is often what you opt for if you've got serious requirements around diverse software stacks, particularly in the presence of accelerators like NVIDIA cards.
None of those things is a silver bullet though. They are among the best tools for that class of job, but while Nix and Bazel both have facilities for coping with multi-platform builds with difficult dependencies, you still work hard for it. Nix/NixOS will get you just about perfect builds on a given OS/CPU pair, and Bazel has a lot to offer there too, but when targeting `aarch64-darwin` for local development on Macs and `x86_64-linux` on the deployment target, it's not a free lunch.
The documentation and OSS community support and stuff like that has gotten better (a lot better) in the last year or two for both of those things, but I still can't in good conscience recommend either to a shop that doesn't already have experienced staff (or staff that is passionate about getting that experience): it's a steep learning curve because the problem is fundamentally insane in practice if not in theory.
For folks that want code to run the same in production as in development and can't categorically commit to languages and libraries that religiously eschew diverse native dependencies but still don't want to go through the looking glass with e.g. Nix the best bet in my experience is to provision dev servers adjacent to the production infrastructure and lean on a massively improved remote development experience in e.g. VSCode compared to even a few years ago.
None of those things is a silver bullet though. They are among the best tools for that class of job, but while Nix and Bazel both have facilities for coping with multi-platform builds with difficult dependencies, you still work hard for it. Nix/NixOS will get you just about perfect builds on a given OS/CPU pair, and Bazel has a lot to offer there too, but when targeting `aarch64-darwin` for local development on Macs and `x86_64-linux` on the deployment target, it's not a free lunch.
The documentation and OSS community support and stuff like that has gotten better (a lot better) in the last year or two for both of those things, but I still can't in good conscience recommend either to a shop that doesn't already have experienced staff (or staff that is passionate about getting that experience): it's a steep learning curve because the problem is fundamentally insane in practice if not in theory.
For folks that want code to run the same in production as in development and can't categorically commit to languages and libraries that religiously eschew diverse native dependencies but still don't want to go through the looking glass with e.g. Nix the best bet in my experience is to provision dev servers adjacent to the production infrastructure and lean on a massively improved remote development experience in e.g. VSCode compared to even a few years ago.