If they give you a valid subpoena, then yeah, you have to give them the video in a format they can use, you can't give them a bunch of encrypted video files and say "Good luck trying to watch it".
If you refuse to hand it over and they get a warrant and seize your NVR, then I'm not sure if they can compel you to decrypt it, but you may already be in jail for contempt and probably are spending hundreds of thousands of dollars in legal fees at this point, so most people would cave long before this and just hand it over.
Not a lawyer, but I read about cases where the judge put people in prison for contempt because they refused to disclose passwords for encrypted content. There was a guy that was in prison for more than 1 year in such a case.
To the extend that I've read, they can demand that they hand over data/documents that they know exist, regardless of whether they're encrypted. It's akin to being forced to open a safe if they have a search warrant for a specific document.
What they can't do is say "decrypt this blob and let us look around".
It depends. If it's against another party they can compel you to decrypt it. If it's against yourself they may respect a 5th amendment claim, or they might not.
If you refuse to hand it over and they get a warrant and seize your NVR, then I'm not sure if they can compel you to decrypt it, but you may already be in jail for contempt and probably are spending hundreds of thousands of dollars in legal fees at this point, so most people would cave long before this and just hand it over.