[nijave]

It's both. Some models, given web search and web fetch may only run a search and assume the summary text is correct and blindly return it. Others will validate by running a web fetch and checking the whole page contents. Even better, the model will run multiple searches and fetches to cross check the information. The best models will attempt to verify whether a source is authoritative or not and try to only return authoritative results.

I have an example here: https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...

Tldr; all the Claude models had identical tools and some used them efficiently and verified data while others did a crap job and hallucinated responses. Additionally, Exa MCP tools generally worked better even on older/smaller model (Llama)

[deno]

Unless I'm missing something I think to a large degree you're just comparing system prompts.

If I add "Research the question extensively" to your prompt at the end I get the correct answer from Haiku and Sonnet Med on first try and I've reproduced the original prompt not returning the answer.

Unfortunately every other run now gets your gist in results.

[nijave]

It could be variations in system prompt but I'm not sure how "change the user prompt to encourage tool usage" proves that either way

[deno]

The point of this was supposed to be "model tool use competency" not baseline over-eagerness.