Hacker News new | ask | show | jobs
by strickvl 712 days ago
I think not. Normally if you get those kinds of errors you wouldn’t get any output at all. In the blog I show that all 724 of the test cases got proper JSON output etc for the queries so I don’t think this was an issue. I think these kinds of topics would have been well covered in the training data, and probably the OSS models would have used similar data so I don’t even think there’s a disparity to be found between proprietary vs OSS models here.
1 comments
resource_waste 712 days ago
>Normally if you get those kinds of errors you wouldn’t get any output at all

I am not sure. I disagree. If there is a pro-chatGPT user, I'm probably it.

Ive often seen it give significantly less effort to answer the question.

link
strickvl 712 days ago
Interesting. I can maybe try finetuning one or two of the so-called 'uncensored' open models and see if that makes a difference. A bit harder to switch out the dataset completely, as that's really what I'm interested in :) I think the general point that finetuning a model for some custom task works is fairly uncontroversial, but if OpenAI's poor performance was on account of these kinds of guardrails it'd be yet another reason someone might want to finetune their own models I guess.
link