[airocker]

Are tools fine tuned in both or just prompt based? For a fair comparison

[fudoshin2596]

No fine tuning. Looks like he's do raw model capabilities with simple prompt. repo: https://github.com/parea-ai/tool-use-benchmark

[Joschkabraun]

All models got the same prompt fed which was essentially "Question: {question}". And then the API's accept the function call definition

[airocker]

But what if there is a bad model that fine tunes with only tools?

[Joschkabraun]

What do you mean with being a bad model?

If the model is really good at tool use, then it will broadly useful as it needs capabilities to generate the tool definition. So, there should be some transferability.

[ec109685]

If the exact tools test was part of the training of the model, wouldn’t that throw off the results and not generalize?