[Xenograph]

That explanation sounds circular to me - by definition any word you'd use in that way is an adverb.

A more intuitive explanation is perhaps that "overseas" came from the combination of two words, "over" (a preposition), and "seas" (a noun). And when you write it out as three words "going over seas" it makes perfect grammatical sense.

Disclaimer, I have no idea if my explanation is rooted in facts about the English language or how it actually developed. But that's how I'd answer the question if someone asked me for an intuitive explanation about why we don't use "to". Because there's already a preposition ("over")! You wouldn't write "going to over seas", would you?