You would provide this file for them to download alongside where you show the email address; the file would be equivalent to an API key - should be a symmetric key. It's sort of just swapping out an Authorization header for a file with an expected checksum/contents.
If the barrier is technical abilities, trying to set up public/private keys and/or signing the emails could hinder adoption.
One more suggestion that would, IMO, be fairly simple. Tell the user to write in their subject line a {accessKey: 'someKey'}. The access key is given to them at the time of creation and can be regenerated as often, or rarely, as the user prefers. If they want update it weekly to provide an extra layer of security they can, or they could keep it the same forever with it just bein a handy backup in case they do link the random email. Just a thought.