If you're using an iPhone (dunno if Android has a similar feature), you can enable "silence unknown callers" in the phone settings. Then anyone not in your contacts or suggested contacts won't even ring.
That's great, but my doctor calls every week from an "unknown number" to check in with me and I have to get that call, I can only assume that's some kind of NHS voip solution for calling patients, so even if I wanted to I can't whitelist their number.
I mean the irony is that when you have a huge internal phone system like this the only way to do it "right" is to spoof the number to something consistent.