If my kids are asleep we turn the TV volume down. Why would you wear headphones with your spouse, you can't have any conversation so it defeats the purpose of watching together
If you want to talk put a headphone in the outer ear, and you can still talk softly to each other. Headphones are super immersive sounding. And if you have the pros, you set them to transparency mode. You can still talk, but kids won't hear the movie.
You should consider turning down your headphone volume if the idea of talking to someone with them on seems far-fetched.
I used to be like that and my hearing shows it, and I've learned that just because I'm listening to something, it doesn't need to immerse my entire aural apparatus.