Consistency. Even if you've never held a telephone receiver, if it means 'call' in one place, it's very likely to mean the same thing in another.
We could be using random hieroglyphs to the same ends, but people seem to always make their own (barring a few exceptions, like the hamburger menu). It's probably a better idea to use something with some grounding in reality rather than make your own from nothing, since doing that is hard, even for actual designers.
It also takes a long time for those purely digital ideas to filter out into the mainstream and get a shared name. For many years I would hear people same, “click the thing in the upper left corner”, “click those 3 lines up there”, or something similar. The term hamburger menu is starting to filter out there to the point where I feel comfortable using it. Even the term, hamburger menu, is a skeuomorphic name for a digital control. No one knows what to call those 3 lines, but everyone knows what a hamburger is. I’ve even seen some sites use a literal hamburger icon.
It also took a long time to standardize on the hamburger menu. Many also tried to use an ellipsis for a long time. Some still do. Sometimes those dots are vertical… is that a thin hamburger or a vertical ellipsis? I heard one person trying to make the term “tots” happen for this style of menu… tatter tots to go with the hamburger.
Contrast that to a “gear” menu for settings. They see a cog or a gear and everyone knows what that is without training, even if they aren’t a mechanic.
Do you need to know the meaning of the latin words manus (hand) and facere (to make) to understand the English words manual (by hand), factory (place which makes things) and manufacture (to assemble)? Do we need new words now that Latin comprehension is dwindling? Not really.
Language works by metaphor, even if the thing you're alluding to doesn't exist anymore.
It doesn't really matter if they haven't used the original physical artifacts. If it looks physical you can figure out how to use it based on your knowledge of other physical objects you have used in your life.
Of course if you display for example, a spin dial like old telephones that has a particularly quirky way to use, them this doesn't apply.
We could be using random hieroglyphs to the same ends, but people seem to always make their own (barring a few exceptions, like the hamburger menu). It's probably a better idea to use something with some grounding in reality rather than make your own from nothing, since doing that is hard, even for actual designers.