My understanding is _a lot_ of stuff in Nvidia drivers isn't _technically_ owned by them. As in they license out A company to make Y thing with Z license. After 20+ years of legacy those weird licenses add up, and no-one feels comfortable with releasing the code. Whether or not that's true? Hard to say... For comparison AMD has been (since 2015) working hard to make nearly all parts of their drivers upstream in some fashion (amdgpu). While that's not a 1:1 comparison there (AMD has less capital than Nvidia) it shows it _is_ possible to free these drivers.
The real answer is that releasing source code means uncovering (some of) their cards. The company, that relies on selling identical hardware under vastly different prices, won't ever do that. Using game-specific speed-hacks to "fix" games, purposefully written to violate standards, is another issue. Especially, when those games were made with help of Nvidia engineers. Why give up such ability?
I also suspect, that they make use of multiple patented technologies, both in hardware and software. When Java has been re-licensed to GPL, one of the most prominent pain points, that caused endless whining on part of OpenJDK users, happened to be it's font renderer. And we all know, that font smoothing is tricky business, and all font-smoothing tech in existence is patended by MS/Apple/Adobe. When you start replacing closed-source code with free replacement, those patented pieces tend to quickly come up — especially when open-source projects go to great length to work around patent issues instead of shoving them under the carpet.