| They did include the methodology in the actual publication[0], the Guardian just refuses to source their statements. AP used the existing tools for showing how people politically align[1] to generate 3000 identities (equally split amongst the 2 largest tools that are used for this sort of thing). These identities were all set up to have 80% agreement with one political party, with the rest of the agreement being randomized (each party was given 100 identities per tool and only parties with seats were considered). They then went to 4 popular LLMs (ChatGPT, Mistral, Gemini and Grok, multiple versions of all 4 were tested) and fed the resulting political profile to the chatbot and asked them what profile the voter would align with the most. They admit this is an unnatural way to test it and that this sort of thing would ordinarily come out of a conversation, although in exchange they specifically formatted the prompt in such a way to make the LLM favor a non-hallucinated answer (by for example explicitly naming all political parties they wanted considered). They also mention in the text outside of the methodology box that they tried to make an "equal" playing field for all the chatbots by not allowing outside influences or non-standard settings like web search and that the party list and statements were randomized for each query in order to prevent the LLM from just spitting out the first option each time. Small errors like an abbreviated name or a common alternate notation for a political party (which they note are common) are manually corrected into the obvious party they're for unless it's ambiguous or aren't parties that are up for consideration due to having zero seats. In that case the answers were discarded. The dutch election system also mostly doesn't have anything resembling down-ballot races (the only non-lawmaking entity that's actually elected down-ballot is water management; other than that it's second chamber, provincial and municipal elections) so that's totally irrelevant to this discussion. [0]: https://www.autoriteitpersoonsgegevens.nl/actueel/ap-waarsch... - in dutch, go to Publicaties. The methodology is in the pink box in the PDF. Samples of the prompts that were used for testing can be found in the light blue boxes. [1]: Called a stemwijzer; if memory serves me right, the way they work is that every political party gets to submit statements/political goals and then the other parties get to express agreement/disagreement with those goals. A user can then fill them out and the party you find the most alignment with is the one that comes out on top (as a percentage of agreement). A user can also lend more weight to certain statements or ask for more statements to narrow it down further if I'm not mistaken. |