Did you look at egoblur? its a lot more effective at face detection than https://github.com/ageitgey/face_recognition granted, you'd have to do your own face matching to do exception.
I'll take another look at EgoBlur to see if it's a good fit, thanks. When I checked it briefly before, I thought it was focused on post-processing (non-realtime) and didn't integrate with matching easily, but it's definitely solid tech.