The hardware usually comes with vendor-specific libraries (e.g. cuda in the case of nvidia) which are based on a specific version of libc, so then you will have to build your entire alternative OS around that version also.
Which is… never trivial. I’d say 25-50% of my career so far has been repeatedly “fixing” clunky deployments of ROS, OpenCV, L4T, CUDA, cudnn, libc, etc. in Docker and Nix. Fun stuff!