[swyx]

can u test it on say who won the 2024 US election

[ghurtado]

I can't really think of a less reliable test for anything at all than making a random guess as to something that had about 50/50 odds to begin with

Easiest Turing test ever...

[himata4113]

ask it 10 times.

[pixel_popping]

MASSIVE ADVERSARIAL x50

[WarmWash]

Usually the labs do some kind of post training on major events so the model isn't totally lost.

A better test is something like "what is the latest version of NumPy?"

[bakugo]

That sort of test isn't super reliable either, in my experience.

You're probably better off asking something like "what are the most notable changes in version X of NumPy?" and repeating until you find the version at which it says "I don't know" or hallucinates.

[czk]

with thinking off and tools disabled:

  Donald Trump won the 2024 U.S. presidential election.

[redsocksfan45]

I thought that one specifically was placed in the default system prompts of basically all providers.