Hacker News new | ask | show | jobs
by gardnr 930 days ago
How is it better than the model from the team that made the dataset? https://huggingface.co/Open-Orca/Mistral-7B-SlimOrca
2 comments

The Intel one had supervised fine-tuning with the SlimOrca dataset, and then DPO alignment on top of that using a preference dataset.

The technique for generating the preference data is what’s so interesting about that one. Instead of having human labelers choose a preferred response, they generated a response from a small model and a large model, and then always selected the large one’s as the preferred response.

I haven't personally tried that one, but on the HuggingFace LLM Leaderboard:

Open-Orca/Mistral-7B-SlimOrca - AVG: 60.37, ARC: 62.54, HellaSwag: 83.86, MMLU: 62.77, TruthfulQA: 54.23, Winogrande: 77.43, GSM8k: 21.38

Intel/neural-chat-7b-v3-2 - AVG: 68.29, ARC: 67.49, HellaSwag: 83.92, MMLU: 63.55, TruthfulQA: 59.68, Winogrande: 79.95, GSM8k: 55.12