[mbowcut2]

I'm not surprised. People really thought the models just kept getting better and better?

[segmondy]

The models are getting better and better.

[giveita]

That's expected. No one will release a worse model.

[sodality2]

Not a cheaper one, or better in some ways, or lower latency, etc?

[giveita]

They do that too but right now it is an arms race as well.

[guerrilla]

Maybe. How would I know?

[jMyles]

...even if the agent did "cheat", I think that having the capacity to figure out that it was being evaluated, find the repo containing the logic of that evaluation, and find the expected solution to the problem it faced... is "better" than anything that the models were able to do a couple years ago.