[mcintyre1994]

> We asked it [GPT-5.5] to assess whether each leader had made a falsifiable claim about the future as part of its main thesis. About 1,400 did. We then extracted those predictions, and asked the AI to mark out of ten both how contrarian the leader’s outlook was at the time and how accurate the prediction turned out to be. We ran those queries several times and took an average.

I understand why it wouldn’t be feasible for a human to do this, but I’m quite sceptical about an AI assessing how accurate predictions turned out to be/how contrarian they were at the time. It seems like that would depend a lot on what sources it chooses, be liable to hallucination or getting poisoned by bad sources, etc. They don’t mention whether they used independent queries for each prediction either, or whether it was doing multiple sequentially.

Given that LLMs can’t really distinguish prompt from instructions etc, I’m sceptical that they can reason particularly well about things like how contrarian a view was at a particular point in time.

[znpy]

> I understand why it wouldn’t be feasible for a human to do this, but I’m quite sceptical about an AI assessing how accurate predictions turned out to be/how contrarian they were at the time.

Quite the contrary actually. The economist has a fairly consistent editing style that gets enforced, so its writing style is very linear and very consistent, fairly easy to “understand” for and advanced llm like GPT-5.5

[flashman]

You assert that, but can you prove it? Likewise I can say that the model is biased towards rating the Economist's claims as 'mainstream' and 'bang on' because as a high-profile publication those claims are repeated elsewhere and hence up-weighted.

I could be wrong - my point is, how would we know?

[znpy]

https://news.ycombinator.com/item?id=26700664

> The reason why the Economist articles all read the same is that they go through a process called "subbing" or sub-editing by the same small group of editors who own the Economist's "voice".

> I interned at the Economist one summer in college and wrote two articles for it - the published articles bore a small relationship to what I had submitted. The sub-editors seemed to be mostly George Smiley-type Oxford and Cambridge PhD's of a certain age with an eclectic range of expertises, as far as I recall.

[platelminto]

I thought the same while reading this - I can easily imagine the AI using hindsight to affect "how contrarian the leader's outlook was at the time", in a way that's similar to how we often do the same.

Would be interesting to do a similar analysis but maybe pushing the AI to search the web for articles/other reporting written during that time, to at least correct for that bias a little bit.

[tgv]

> We ran those queries several times and took an average.

The whole thing is bizarre. They could at least have drawn 100 samples and evaluated the model's response. But no, they ran it a few times and hoped that the slight randomness in sampling magically resulted in a balanced assessment of accuracy and contrariness, at some point in time for which we don't even know how much training data there was, nor if that data accurately reflects the opinions of that time. But hey, we've got a graph, so it must be true.

How can people be so stupid?

> cheap bioethanol ... [has] yet to bring about the breakthroughs we prophesied.

Oh.

[memoriyato3]

you can still judge if it was correct or not, irrespective of contrarianness, still a good signal to measure

[throwaway27448]

It can give a judgement, correct or not.