[meric]

You're right a lot of the loanwords and similarities between the languages could and likely happened later on.

I think there's no way Cantonese and Vietnamese don't have lots of connections with the ancient Yue people; A whole people don't just fall off the face of the Earth, unless there was a most serious kind of genocide. Almost certainly Cantonese and Vietnamese can trace some of their ancestors to the Yue one way or another. It's true Chinese at the time called everyone south of them Yue, but it didn't mean they were that much fragmented into tribes. The kingdom of Nanyue has borders pretty much exactly matching Guangdong + Vietnam. https://en.wikipedia.org/wiki/Nanyue. Nan = south, and so does the "nam" in Vietnam. I don't think that's a coincidence.

But your larger point is very much correct now that I think about it - the history of vocabularies can be very much complicated.