I would suspect during pre-training. Even before RLHF was a thing models exhibited this bias. My bet is that it has to do with the training data corpora being composed in large part of left leaning content, maybe from social media platforms like reddit.
I wouldn't minimize the potential contribution of the teams building + evaluating the models, either.
If you have an entire office full of "lib-left" hackers, executives, et. al., their biases are going to have an effect on the models they build, irrespective of source material.
As an example: there's a lot of well-established science in support of anthropocentric climate change. And yet, interpretations (or even acceptance) of that science have become political dogma.
I think it's probably safe to say that engineers broadly trust science, whatever their personal views on social or economic policy. So models that are trained and evaluated on following scientific consensus will easily be tagged as "left-leaning."
It's because most people are lib-left. The Financial Times had a great article about comparing social media vs AI. https://archive.ph/Q61eT
It's no surprise that the models would reflect most people's feelings. However what FT showed was that models are shifting the center of gravity towards the right. Their chart showed people are on a bi-modal distribution where most are on the left and some are on the very right, less in the middle.
Then they compared AI and showed how it moved the center of mass towards the center which is moving it towards the right (since most of the center of mass is on the left).
In other words, models are providing answers to the right of where most people would be.
There are no facts and reality when the questions are "Closing the borders may be necessary to preserve a country's cultural identity" and "Those who have money should be able to send it abroad without asking the government's permission", etc.
It's a matter of opinion and has nothing to do with "facts".
In a perfect information world, the first question could have a factual answer. It's not asking if preserving the cultural identity is worth doing, it's asking if closing the border is necessary to do so.
Of course, people are pre-primed to read it as "should we have more or less immigration".
1. The facts in that case would be that capitalism is designed around having consumers who can afford things by working. Welfare (including pension) depends on a large number of people funding a smaller number of people's lifestyle. Giving rights and education to women leads to a falling fertility rate which leads to an aging population. Most developed countries are at the point where they are dependent on immigration to maintain the current system - or need to make drastic changes, like letting older people starve and die, in order to survive without immigration.
2. Those who "have" money made it by relying on a society that was built with other people's money - roads, education, health, utilities, etc. It is only fair if they contribute to the continued running of that society instead of concentrating the wealth offshore and still wielding the disproportionate increase in power as a result.
Of course, it's much easier to go "REEEEEE, YOU HATE AMERICA! YOU COMMUNIST!" But bots have to be trained specifically for that, which tends to gimp their intelligence to match the intended audience.
The bias creeps in when they pick what sources go into the dataset.
The second biggest factor is RLHF because that's the point of it. All LLM providers aim to make models "safe" and safety in our modern discourse means not expressing ideas that contradict the "left ideology".
Safety means protection from liability. That's it.
It makes sense that a liberal bias emerges when you're looking for answers that are from professionals and not homeopathic cranks. Especially in fields like medicine, programming, engineering/design, etc.
Is it truly bias though? As others have pointed out in the thread the list of things considered left right now are essentially arbitrary. "Woke" has been applied to basically anything and everything that a particular party disagrees with, so much so that it's effectively meaningless at this point.