[nico]

Just did a test last week and OpenAIs research was way better. Found 10x more sources and did an overall pretty great job

The task was to lookup information about a late distant family member who had been a prominent employee in a certain foreign government about 100 years ago

Gemini barely scratched the surface and pretty much gave up

ChatGPT on the other hand, kept building up on its research, connecting the dots and leveraging each bit of acquired information to try to find more

[consumer451]

Would love to see this repeated with this latest version from Google.

Man, what's really missing from all of this is a 3rd party AI Consumer Reports type site for all of these LLM tools. Whoever does this thing that does not scale will have a highly referenced site on their hands.

[jeffbee]

Throughout the entire 20th century the main determinant of a Consumer Reports rating for a car was whether you could put a wheelchair in the trunk. Hopefully the AI agent industry does not sprout a similarly worthless metric.

[consumer451]

I almost didn't use that as the comparison for their lack of rigor, but it gets the idea across.

[shigawire]

Isn't that what llmarena does?

[consumer451]

It tries to in a way that scales easily, and is also easily gamed.

I want a staff of human testers, each with domain expertise. If the goal is to replace humans, should there not be a real human metric?

I want a physicist asking their battery of physics questions, 4 different kinds of devs asking their battery of dev problems, a couple chefs asking for cooking techniques, etc.

Now on to "Deep Research," 6 different kinds of OSINT/secondary analysts who ask new problems each time, and compare it to their days of human work.

We really need this as a species, otherwise the brain dead C-Suites of the world are going to keep buying the hype, which is often very premature. This could have real consequences, and it apparently already has.

It's insane to me that we are investing, what, almost $1T into LLMs, and have not spent the ~$1.5M/year to do what I described above.

[consumer451]

^ I really should have used the "myopic," instead of "brain dead" to describe the C-Suites of the world. My apologies.

[SequoiaHope]

I suppose Consumer Reports could do it!

[phonon]

But this is a brand new version? Why not run it again on Gemini 2.5 Deep Research mode and report if it's better?

[infecto]

I do think they are leaps in front of everyone else from the product perspective and everyday its looking more to be the battleground where money is going to be made.