As someone who has had a home ML server since 2016 with two TitanX GPUs, and has worked on and maintained numerous servers since then I can definitely echo that maintaining Nvidia drivers, along with CUDA, CUDNN, etc has always been a hassle. It's certainly gotten better over time, but it's still quite fragile.
What about trying something like enabling forward compatibility for CUDA using an older driver?
https://discuss.pytorch.org/t/torch-is-unable-to-detect-cuda...
(This issue was actually just posted within the last day, so clearly people still have problems.)
If you haven't run into any issues, then I'd say you're very lucky. Just don't pretend lots of others haven't run into issues.
https://forums.developer.nvidia.com/t/545-drivers-have-bad-f...
https://forums.developer.nvidia.com/t/wayland-native-wayland...
https://gitlab.freedesktop.org/xorg/xserver/-/issues/1317
and literally me being a Linux Nvidia 1080ti user for years and having plenty of issues
https://i.imgur.com/lwHDZoq.png