[mrhottakes]

The constant issue with these sorts of categorization efforts is that the outcome is entirely dependent on how the responses to "politically charged questions" are graded as left vs. right. You're mostly just examining a delta in biases between the model and the investigator.

[raxxorraxor]

Yes, but I think it is still a viable metric to some degree. I wondered about Gemini being dead center here. At first it was obvious that it was actively trained to give biased responses to anything controversial. It was deservedly made fun off because it tried to warp reality. I still don't trust it today, although that is pretty much true for any model.

[platz]

The alternative is a High-Dimensional / Embedding-Like Approach where question responses aren't tied to fixed axes, but rather the full response set is treated as a point in latent space.

Then it's on the researcher to examine the clusters and assign labels. There's also not a nice mapping that's a-priori interpretable in low-dimensional pre-existing axes.

Probably only used in research than consumer websites, under more controlled conditions; there are very few public political tests doing this transparently

[groundzeros2015]

Exactly. These models don’t hold coherent views. You can prime any of them to agree with any view.

[hogehoge51]

Your multimeter reads out voltage relative to the black terminal, it's your responsibility to find the ground plane.

[mrhottakes]

My multimeter doesn't need me to tell it how much a volt is or feed it subjective measurements of what resistance means

[MarkusQ]

Resistance is when you're nobly standing up to the other side and things get a little out of hand. Domestic terror is when the roles are reversed.

[mektrik]

Fair; have tried to combat this issue in a few ways.

Each model's position is scored against outside political-science data (Chapel Hill Expert Survey for party positions, World Values Survey for where populations sit).

The stance coding is done by a separate model with a published prompt + a second model from a different lab re-scores a sample and we publish where the two disagree.

So not perfect but (as far as I can tell) one of the more defensible approaches.

[selfmodruntime]

This is especially apparent in the 'worldview' sorting under the `bias` section, which lists the German FDP to be further right than the CDU (which is nonsense) and also barely registers the FDP as libertarian when they are a free speech, small government, personal responsibility and free market party. They also register "Die Linke" as Libertarian-Left, which could not be further from the truth. "Die Linke" barely has libertarian values at all, being pro state-governed economy, having an ultimate goal of democratic socialism and they're certainly big government. They're also leading a large deposession effort for large landlord companies. I'd honestly put them into "Auth-Left" territory.

So yeah. The bias is a bit nuts and you could reasonably accuse the study/report of misdirection/misinformation and plain fasehoods.

[mrhottakes]

Yep. All studies like this are just measuring "how much does the model agree with my preconceived notions?"