[falcor84]

I feel a bit bad bringing this up, but should Gemini actually be considered SOTA?

They make impressive demos, but I can't recall any of their released models being at the top of any leaderboard.

EDIT: Sorry, looking into it a bit more now, they still seem to be at the top in term of the context window, so they got that going for them.

[azinman2]

Leaderboards are misleading. Try diff models for YOUR task and you’ll see a wide variety of outputs compared to “official” rankings.

[falcor84]

Ok, maybe I haven't experimented enough; so for which tasks is Gemini the SOTA?