Voting Tory is a minority position however you count it: no government has had even 50% of the votes cast (let alone the votes of those eligible to vote) for decades. Brexit meets the first 'majority' criterion, but not the second.
It is entirely possible to be pro-Tory and pro-Brexit while being anti-xenophobia. But the Conservatives have been carefully-mining the anti-foreigner vote for decades (as did New Labour) and it's not plausible to argue that Brexit would have won without that sentiment: it was too large a part of the expressed reasoning of Brexit voters as a group when polled.
The real loss here has been the AV referendum in 2011. It blows my mind how dense the electorate was for keeping the FPTP system.
Now the Tories have realized that they can spew bullshit lies and win an election with minimal consequences. They also have discovered how much easier it is to game the damn failing electoral system. (e.g. in Cam I have seen Tory pamphlets pushing for Lib Dems or Labour just to split the left vote).
New Labour mined the anti-foreigner vote? Are you actually from the UK? New Labour was famously pro open borders. They deliberately opened the immigration floodgates as much as possible, for instance, one reason there was so much immigration from eastern Europe to the UK when Poland etc joined the EU is because the UK was one of only two countries that didn't immediately utilize the 'transitional controls' mechanism to keep immigration restrictions in place. Despite the EU's rhetoric about open borders and such at the time, in reality most countries didn't want to be flooded with cheap ex-eastern bloc labour. UK under the self-proclaimed Labour party did.
Notoriously, a few years ago a former advisor to Blair went on record to say that New Labour deliberately encouraged as much immigration as possible in order to "rub the Right's nose in diversity", and:
"He said Labour's relaxation of controls was a deliberate plan to "open up the UK to mass migration" but that ministers were nervous and reluctant to discuss such a move publicly for fear it would alienate its "core working class vote".
It's really difficult to reconcile this history with the ideas you're expressing above. Far from "carefully mining the anti foreigner vote for decades" they were explicitly undermining it by hiding the true extent of what they were doing.
Yes, they pursued migration because they considered it to be economically beneficial. But their electoral coalition also included groups which disliked it - retirees in particular, and some parts of the older white working class vote too as you note. So the action to increase 'good' migration was never really openly argued for as a public good (your quote is from an advisor, and would never have been released at the time) while being matched with Mail-placating rhetoric on asylum designed to separate the idea of deserving and undeserving migrants and demonstrate 'getting tough' on those who were sufficiently unpopular. There was a huge increase in the use of immigration detention, for example, and the deliberate destitution of many asylum seekers as a means of appearing tough. As with many New Labour policy areas, the aim was not to get the policy 'right' and leave it, but to keep the issue live to generate a steady series of headlines indicating that something was being done. Criminal justice policy was similar, with more than one reform bill a year for most of the period of the government.
So overall, I don't see deliberately increasing migration and deliberately using dehumanising rhetoric and policy changes against (some) immigrants to shore up electoral support as being at odds. The current government is strong on anti-tax rhetoric whilst presiding over heavy tax rises. Labour wanted immigration to keep wages low and increase growth. It also wanted the votes of people who wouldn't want that. So it did the former fairly quietly, while performatively stoking up the Home Office's capacity for villainy. So is modern politics.
(FWIW 'Floodgates' is itself somewhat dehumanising, I think: people aren't an undifferentiated mass threatening to overwhelm the boat, they're just people. 'Gates' would have done just fine.)
Ok, maybe the assumption of xenophobia is unwarranted, but restricting of immigration was one of the main selling points of Brexit.
Also, the point about age holds - under 45's voted heavily in favour of Remain (as did London, Scotland and Northern Ireland)