[m_w_]

It's a bit disheartening to see no comparison to other models here - and I'm not sure this pushes the curve anywhere. 3.6 flash is more expensive than GLM 5.2 - but seemingly worse, although this post is really light (lite?) on details.

It seemed for a time that Google had finally gotten the ball rolling, but I'm doubting that more and more as time passes. We'll see what happens with 3.5 pro I suppose.

[XCSme]

Here, my comparison of 3.6 Flash vs Sol vs Luna vs Terra: https://aibenchy.com/compare/google-gemini-3-6-flash-medium/...

[jdthedisciple]

How does your comparison work? It places Gemini 3.6 Flash Medium above GPT 5.6 Sol High and Fable 5 Medium, which makes me skeptical because that... would be making headlines that I'm not seeing right now.

[XCSme]

I have created various questions/tests and put the models through the same tests.

I record whether the answers are correct, and the generation stats (costs, latencies, tokens used, etc.).

I have no idea why the Gemini models do so well.

I have recently added new tests, whose sole purpose was to find some cases on which Gemini 3 Flash fails (I don't like cherry-picking models or tests, but I also find it strange Gemini Flash models leading in accuracy). I made a more complex coding/tool-usage test, that I expected it to fail, it did fail it once locally in my debug tests, but when I finalized the test and ran the entire testing suite for all models, somehow Gemini 3 Flash still got it right...

Gemini models are REALLY intelligent (and they are actually my favorite model to use via the chat app to ask questions), but they somehow fail in real-word coding tasks where they have to modify files, check results, debug, etc.

My tests harness provides a lot of mock data, and limits the number of actions a model can choose from. I am starting to think that maybe the models are not bad, just that the coding harness are not optimized for those type of models, and Google doesn't really provide their own "Codex".

[XCSme]

Oh, and I've also added weights to different categories, so Coding and Tool usage categories influence the score more. This done both to better account for how most people are being used, and also to reduce Gemini's dominance in general/domain specific knowledge.

So yes, Gemini models are at the top, even if I actually (not proud of it) tried to make tests that actually favour other coding-focused models.

[jdthedisciple]

Interesting, well it'd be interesting to check out some individual examples where Gemini beat the others.

Also, would be great if you could add GPT 5.6 Sol XHigh and Fable 5 High as well, just to see if at least those beat Gemini which is currently your #1.

[XCSme]

I don't like to divulge tests, but one of them is a chess puzzle.

> would be great if you could add GPT 5.6 Sol XHigh and Fable 5 High as well

I would like too, but I avoided them for several reasons:

1) Cost - this is a hobby project, those models would cost tens of dollars for each benchmark run, multiply this by tens or hundreds of models and ...

2) Time - the high models are already taking a really long answer to respond (5-10minutes per question). I run each question with 3 repeats (run the same test three times), so it would take 30 minutes per test. If I change my tests, methodology, or add a new test, it would take a really long time to run the benchmark. Also, I like having results immediately when a new model is released, now I can post within 30 minutes of a model's release the benchmark results.

3) High reasoning usually does WORSE on most tests - if you look at the leaderboard, it's sometimes counter-intuitive, but models with high or max reasoning usually do worse than medium and low. This is because the questions are quite targeted/direct, and the models overthink the question and miss the solution. Or the long thinking context makes them perform poorly. The generation tasks (SVGs/HTML animation) are usually better with longer reasoning, but short code fixes, trivia questions, puzzles, etc. are answered by low/med reasoning with more accuracy in general

Also, Fable is borderline un-testable, it refuses to answer many questions, so it scores poorly anyway.

Gemini scores 21/22 because it answers all tests, and it does them correctly, consistently. The only failed test is I think because it miscounted the lines in a file, when responding on which line the bug was in a code snippet.

[dudeinhawaii]

You should really provide more on your methodology because as it stands, it really doesn't pass the sniff test. GPT-5.6 Sol on Low beats Fable Medium by 10% and Gemini-3.6 Flash then beats them both? Fable is number 20?

This does not match any lived experience or developer experience.

It'd be helpful to know _what_ you're testing and break that out by dimension. You mention randomly selected questions. How does that work?

With n=22 and binary pass/fail, the 95% confidence interval on a pass rate spans roughly (+-)15-20 percentage points. There's just not enough data ironically, for this leaderboard to mean anything. Ranks #5 through #25 are statistically indistinguishable

[XCSme]

Thanks for the feedback, really good points!

There is some short info about the methodology here: https://aibenchy.com/methodology/

> GPT-5.6 Sol on Low beats Fable Medium by 10% > Fable is number 20

Fable loses a lot of points because it often refuses to answer questions. Asking a basic tool-usage challenge, Fable responded with refusal: "This request triggered restrictions on violative cyber content and was blocked under Anthropic's Usage Policy. To learn more, see https://platform.claude.com/docs/en/build-with-claude/refusa...." Even in practice, you ask Fable something trivial, and it refuses to respond. I think the score accurately represents how the model is behaving in real-world usage.

> Gemini-3.6 Flash then beats them both Gemini models are the most intelligent overall. The tasks are not coding-only. Gemini excels in general knowledge and domain specific knowledge. Gemini models, even old ones, still top many charts on specific use-cases[0][1]. Depending on how you weigh those cases, the leaderboard order can vary quite drastically, as some models are very strong in some domains and weak in others.

> You mention randomly selected questions. How does that work? Randomly selected, means I have manually created the questions/challenges to span across various domains and agentic surfaces. Questions vary from coding tasks, tool usage, trivia questions, chess puzzles, car-wash-like challenges and more.

> With n=22 and binary pass/fail Each test is run 3 times, so in total we have 66 tasks. Also, apart from correct/wrong answer, the final score also includes the pass rate for each test (how many out of three attempts), how good the reasoning is (they have a hidden reasoning score where available) and other small factors. Also, some tests in some categories involve a series of tasks/requirements (i.e. implement this function, call it, do some processing on the result, combine the result with some built-in knowledge data, etc.).

I do agree that 22 tests isn't that much, and I'm slowly adding more, but even without the leaderboard part, the comparison feature is what's I think is most useful. You can see for the exact same tasks, which models do better, which do it faster, which cost less, etc.

> _what_ you're testing and break that out by dimension There is a category breakdown for the test results, so you can see and which sort of tasks models fail.

Everything aside, when you manually ask a model to test its capabilities, I don't think it takes many questions to realise how good/bad that model is. Sometimes one prompt is enough, you ask it to do something, and see how it reasons about it, how fast it does it, how efficient the steps are and how good the result is. Yes, the performance may vary across tasks, but I'm pretty sure if you did a blind test with a chatbot, you could easily realise how good the model is in just a few questions/tasks.

I think no benchmark is perfect, mine is far from it, but it's simply another different, independent data-point. Apart from that, I made this for myself, and I'm using it myself. I don't trust that all popular benchmarks are not in the training data, and I think many benchmark the wrong things which don't correlate to how I use the models day-to-day myself. I just made the results publicly available, in case any one else benefits from it. I've probably spent thousands in LLM costs, and probably more than 100 hour building this, without benefiting in any way from it (outside of the joy of building it and me using it personally to compare models); as long as models cost stays reasonable, I'll keep building it and test new models as soon as they are released.

[0]: https://x.com/browser_use/status/2079602472516264010/photo/1 [1]: https://artificialanalysis.ai/evaluations/mmmu-pro#mmmu-pro-...

[armarr]

GLM was twice as verbose running the Artificial Analysis benchmark. So it ends up being more expensive

[Havoc]

>verbose

GLM defaults to max effort btw

https://docs.together.ai/docs/glm-5.2-quickstart#reasoning-e...

[maxloh]

Not really. Gemini 3.6 Flash actually cost $0.01 more per task, compared to GLM 5.2.

https://artificialanalysis.ai/models/gemini-3-6-flash

[mdasen]

But to run the entire benchmark it cost $727 with Gemini 3.6 Flash and $925 with GLM-5.2, $198 (21.4%) less. I tend to look at the cost to run the whole index rather than the weighted average cost per task.

[CSMastermind]

All the benchmarks I see put it around the capabilities of Opus 4.8 Medium or Sonnet 5 High.

As far as I can tell it's slightly better than GLM 5.2.

[zwaps]

according to AA it's not better than GLM 5.2 and that's surprising to me

[CSMastermind]

I personally take AA with a giant handful of salt.

[florakel]

It’s really surprising. When Apple announced the multi-billion dollar deal with Google to power Apple Intelligence I thought great things were coming. Instead we are getting more and more bad news: delayed Pro models and AI leadership leaving. I wonder if Apple know something the rest of us don’t know or if they are already regretting their decision.

[WarmWash]

Besides Apple apparently making Siri AI model agnostic, the choice to go with Google was almost certainly for practical reasons. Google is a low-risk established player that already has a long work history with Apple. Google also isn't in an existential battle to establish themselves, Gemini still amounts to just another project at Google. There is tangible non-zero risk that either OAI or Anthropic will be gone in 5 years, or will be forced to leave Apple high and dry to save themselves. There is almost no risk Google will be in either such position. And worst case scenario, Google has incredibly deep pockets should Apple pursue a "refund."

[stingraycharles]

They also famously hate nvidia since 2008 and would prefer to use TPUs, at least historically

[nateb2022]

> Besides Apple apparently making Siri AI model agnostic

Ridiculous statement, pretty much all LLM tooling is model agnostic.

[revolvingthrow]

What Apple wants out of Google is Siri that runs at 8gb ram and isn’t a horrible embarrassment that feels like a primitive markov chain. Given how good Gemma 4 is, Google can squeeze some serious performance in small models. Whether they can make bleeding edge models is irrelevant to Apple.

[zarzavat]

"Siri, please solve the Jacobian conjecture, and also set an alarm for 8am tomorrow"

[Petersipoi]

As someone on the Apple beta.. the model is almost completely irrelevant to the experience. Apple has gone and done Apple things by nerfing the experience so completely that almost any model in the past year would be fine. I still reach for ChatGPT/Claude/Grok constantly instead of the AI toy that Apple calls the new Siri.

[winstonp]

Apple isn't counting on their model to be a frontier coding and cowork model. Gemini is perfectly fine for the tasks that new Siri is supposed to be doing.

[Oras]

The bar was quite low with Siri that adding any model would make a huge difference.

[ur-whale]

> It's a bit disheartening to see no comparison to other models here

Disheartening, but not surprising: the comparison would not be very flattering for Google.

[sczi]

The one thing I've found google's models to be the best at is proofreading text in non-english languages. Probably because I imagine they have the most training data for it as Google probably has the most complete archive of the internet.

[Computer0]

My usage of GLM 5.2 has been defined by slow throughput and flaky providers, in many (definitely not all!) applications a dumber, faster, and more consistent model makes more sense to me.

[dd8601fn]

> really light (lite?) on

Light. Lite is product marketing seepage.

[crab_galaxy]

Yeah that’s the joke :p

[kzrdude]

I've never questioned the word lite before because it's existed my whole life.. So does it make sense? Why does it exist and where does it come from? More than coming from "light".

[Alpha3031]

There's an Merriam-Webster article on it: https://www.merriam-webster.com/wordplay/lite-word-history

[dd8601fn]

Sorry, it went right over my head!