You're right I do fail to consider this, as it's not really an alternative to Shor, because annealing is horse shit that nobody can decide the computational complexity of. Not even D-wave thinks it might be. [1]
I'm not spreading FUD; I am correcting misinformation from muppets whose understanding doesn't go beyond press releases. Nobody has yet done a Shor factorization of the number 15; the end, and even if someone's press release says so there is no scalable way of factoring large integers.
[1] https://www.nature.com/articles/s41598-018-36058-z