Bad actors exist and they're cleaver. Plus these days you don't have to remember anything. Adding an updatable PIN field to a personal call file in such an app coupled with a software protocol with a pop-up button labeled say 'PIN entry' which appears once a contact device answers (say, with a message enter PIN) that can transmit the PIN number would be sufficiently user-friendly for such an app. Better than receiving junk calls while waiting indefinitely for regulations with teeth from dysfunctional legislatures.
Plus how would we share this "pin"? Maybe we could append it to the phone number, like "+1 (616) 281 - 2123#32343" - Oh wait, it's already a number, what's the point of adding more onto it? It would literally turn into the same system that we have, as you have to share the pin with everyone anyway.
It's different because then not every "number" will result in a call being placed. Determining whether a number is valid requires making a call, which ties up robocallers.
You could have a Captcha to force that the person on the calling ewnd is a person and not a computer. That would increase the costs for telemarketeers.
Or the system could ask for information that someone who knows the recipient might know but a telemarketeer might not know, e.g. "What is the 2nd letter of my first name?"