[topicseed]

Gemini 1 Ultra was also said to be on par with ChatGPT 4 and it's not really there so let's see for ourselves when we can get our hands on it.

[reissbaker]

Ultra benchmarked around the original release of GPT-4, not the current model. My understanding is that was fairly accurate — it's close to current GPT-4 but not quite equal. However, close-to-GPT-4 but 4x cheaper and 10x context length would be very impressive and IMO useful.

[refulgentis]

No, it benchmarked around the original release of GPT-4 given 32 attempts versus GPT-4's 5.