caret notation just maps all ASCII control characters to ^ + letter, many of which do not always have keys. "Backspace" is 0x08 = 8th letter is H = ^H
it's also not just used as an input, but also what a terminal that can't do or doesn't understand a backspace as "delete character, step to the left" prints instead.
But like I said, it's not just letters, if you use `cat -v` to show all non-printable characters in this notation, you'll also see things like "^@" for the 0x00 null byte and "^[" for the ESC. All of the terminal color-code control sequences start with ESC, so if you pipe colorized output to `cat -v`, you'll see a lot of "^[".
I don't think that's the answer to the backspace=^h question, but HJKL is precisely an artifact of the keyboard that vi was first used on: https://en.wikipedia.org/wiki/File:KB_Terminal_ADM3A.svg / https://en.wikipedia.org/wiki/Vi#Creation - in addition to HJKL being labeled as the arrow keys on that keyboard, also note the far superior placement of escape and control where modern keyboards typically place tab and caps lock; this also made vi a lot more sensible.
(And of course that last bit is a whole topic unto itself; to this day, caps lock is the dumbest key on the whole keyboard to me - mostly useless, takes up prime real estate, and 100% worth remapping on every machine I control. But that's a rant for another day:])
Because the h-key is in the middle row of the keyboard, meaning relatively shorter keyboard distance needed to execute that common command, thereby saving typing time.
it's also not just used as an input, but also what a terminal that can't do or doesn't understand a backspace as "delete character, step to the left" prints instead.