[martinald]

Yes agreed - I wrote this up a while back https://martinalderson.com/posts/whats-going-on-with-gemini/

My view then was they are optimising the models for inference ability on their own hardware AND use cases, which is often speed and time to first token.

They've somehow seemed to end up with terrible compute shortages, which again is surprising given how good Google is at infra deployments AND have their own hardware. From rumors out there they are turning down enterprise deals for Gemini because they don't have the compute.

The problem is they're falling further and further behind on frontier class on coding especially, and since I wrote that article it's got even worse with open weights models undercutting them on price AND intelligence.

[IgorPartola]

They include an LLM response with every single Google search, whether it is warranted or not. That scale is, my guess, many orders of magnitude higher than what OpenAI and Anthropic serve. And for Google none of these are paid interactions since their LLMs do not (YET) insert ads into the responses.

So my guess is that Google will continue having compute shortages until the Gemini enshittification starts.

[martinald]

I don't think so. According to some very basic research there are around 8bn searches a day, or 250bn a month.

Let's assume Google serves AI overviews on every SERP (they don't) and don't cache them (they do, afiak).

And let's assume that each AI overview is 2000 tokens (blended input/output), that's 500T tokens a month.

It's rumoured that anthropic is serving somewhere close to 10Q tokens a month.

Now it may be that AI overviews uses vastly more tokens than that per search, but I doubt it based on speed to render the overview.

My very rough napkin math on this is that maybe AI overviews is consuming 100T tokens/month max (after adjusting for caching and SERPs that don't have them), which would be 1% of Anthropic token volume.

[imtringued]

Well I asked the google AI mode thing what it thinks about your comment and it told me this (edited obviously):

"10 Quadrillion tokens a month means: 333 Trillion tokens per day and 3.85 Billion tokens generated/processed every single second, 24/7."

"At an incredibly cheap, subsidized infrastructure cost of $1 per million tokens, serving 10 Quadrillion tokens would cost Anthropic $10 Billion per month ($120 Billion a year) just in inference compute."

It also had this to say about how google's AI overview works: "Google doesn't just feed the LLM your 5-word search query. The system scrapes the top 10–20 web results, feeds thousands of words (tens of thousands of tokens of context) into the model, processes it, and then outputs the result."

Oh, and it does all of that in less than two seconds. Honestly, whatever Google is doing with its infrastructure is so far ahead of everyone else, I can't believe you fell for such an obvious lie.

There are also extremely obvious holes in your comment:

>Let's assume Google serves AI overviews on every SERP (they don't) and don't cache them (they do, afiak).

Try it out for yourself. Add a few random letters or punctuation. They cache nothing.

[vikramkr]

The tokens served number might include cache tokens which are a huge chunk of agentic token spend - and even with that the estimated burn rate for anthropic doesn't seem wildly off? They spend 1.25 bn per month on their deal with SpaceX alone.

They definitely cache results - I've searched and re-searched an identical query back to back a few times and seen identical results from overview. They are definitely throwing a stupid amount of compute towards these ai results nobody is paying for - changing punctuation and stuff does get you a different response - but they're not doing no caching.

Certainly what they're doing with their infrastructure is impressive but it's not super meaningful at the end of the day for a for profit company to be really impressively good at burning tens of billion dollars on a service nobody pays for while the same tech from their competitors is quickly becoming one of the largest spend categories for many software engineering teams