[hmokiguess]

You should publish your evaluation set, that seems pretty interesting!

What’s your favourite one?

[iainmerrick]

Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.

[hmokiguess]

I thought open benchmarks helped, sorry, guess I was being naive.

[iainmerrick]

Ha, sorry, I was a bit brusque there.

Open benchmarks do help, but they mostly help the vendors, not we the users!

[Espressosaurus]

Keeping tests private is the only way to keep them valid.