I can disable the on-board sound card in the BIOS settings on my Thinkpad (T440p),
so unplugging the USB headset should do the trick (but then there's no music either).
I do something similar but it doesn't help me with the built in microphone in the laptop. I'd have to disassemble the bezel and possibly snip a wire. It's much easier to Macgyver disable a webcam than a Microphone, even if you've got one of those webcams that can see through plastic with infrared. Best I can do so far is disable the device in device manager.
Although I wonder if the bios on this HP let's you disable it...
This. I do my work on a clamshell laptop, got a similar gadget (bus powered, with LED indicators) when I got an external webcam.
Windows/browser permissions don't have device level granularity AFAICT, so I can't allow access to only an external cam/mic, but I can disable the internal ones in the OS for full tinfoil hat compliance.
Most importantly off is actually off when a button cuts the power.