[stephantul]

Sadly 100% generated.

I think the idea is interesting though, although I wonder if training time for LoRA is such a bottleneck to deserve its own, extremely narrowly scoped, leaderboard. Maybe if it was more tasks or more models we could hope that it transfers? With a single task, and a single model, I’d be afraid of this overfitting pretty heavily.

For NanoGPT, I think the idea always was that the ideas can be transferred to much larger models, or serve as stepping stones for investigations on larger models.

[Vineeth147]

That's fair on both points. Much of this was built with AI, but the runs and numbers are real. They are also reproducible, so I would prefer to be judged on that. And yes, using a single model and task can lead to overfitting. The plan is to add more tracks, including bigger models and other tasks, so a technique only matters if it transfers. Right now, it's just the initial track, so your concern is valid. Thanks for the feedback.

[Aerialoo]

AI generates or not - irrelevant if the substance is high quality. Always good to see projects. You should consider hosting this leaderboard on hugging face, you will get a lot more AI enthusiasts and practitioners there.

[Gisbitus]

I understand the feedback in the second paragraph, however I do not understand why we're judging projects by whether they've been AI generated or not.

Have we stopped treating software as a black box? This behavior will only lead to devs moving away from OSS to avoid the AI stigma.

[stephantul]

Not the software, the whole thing.

As an author: show me why you thought this was interesting and why you’re doing it, and why you think it’s relevant. What does it build towards? What does climbing this leaderboard mean to me?

Absent those things, this is just some thing my opus could generate as well.

[Vineeth147]

I fine-tune small models with limited resources, but it's hard to tell which speedup claims are real. Each method, like DoRA, rsLoRA, Unsloth, or packing tricks, uses different models, data, and hardware in their reports.

This makes it impossible to really check their claims in practice.

The only way I've seen these debates settled is by using a fixed task and having someone act as a referee. That's what nanoGPT's speedrun did for optimizers.

The goal is to make a public record that shows which training tricks really save time and which ones don't work when tested again. Each record should explain how the method works, so over time, the leaderboard becomes a kind of lab notebook.

As more records are added, the leaderboard serves both as a ranking and as a detailed log of what was tried. I agree, and that's my next step: adding a second track with a different model family and task (SmolLM2 + SQuAD), so we can actually test if these methods transfer instead of just promising they do.

Honestly, your opus could probably build the framework too. But a leaderboard is more than just code. It also depends on someone being willing to review submissions for cheating, make decisions when things aren't clear, and keep adding new records even months later. That part can't be automated; it needs to be maintained by people.

[stephantul]

Nice! That sounds a lot more like a mission statement than your actual readme.

[nomel]

Put something like this in the readme, ffs.

[Vineeth147]

Done — the why section is now basically that comment, first person, no filler.

[g-b-r]

> Have we stopped treating software as a black box?

When did we begin to? Why?