[usrusr]

My attempt at skimming for "what would be needed": controlled input specifically designed to make the process with the keys speculatively fetch or not fetch address lookalikes depending on key bits, and some observer comparing timing either of fetches to canary addresses after the key has or has not triggered a fetch, or observing how the timing of the crypto parts changes with our without canary fetches beforehand. Or perhaps even outside observability from inputs that would either fetch the same canary address twice, or two separate address, depending on key bits?

In any case, the stack of "this could not possibly be workable / but with enough statistics it can, and computers are plenty fast to generate statistically useful case numbers" is truly mindboggling with these attack vectors.