[rao-v]

We really need to band together to fund / sponsor targeted inducement prizes (a la Nobel laureate Michael Kremer) for open models.

Every 6-12 months, give out $200K to the first model to hit a min threshold on a set of ~5-10 hard benchmarks (+ perhaps one secret benchmark) using a total of 16GB / 32GB / 64GB / 128GB of VRAM (at a min context length of 200K), then move the threshold up. Quantization etc. is dealers choice, it just needs to nail the benchmark on a reference machine by using exactly that much VRAM (no mapping to RAM / disk etc.)

You could crowdsource the funding, and cross subsidize by adding targeted prizes focused on corporate needs (the classic one is PDF processing benchmarks), and say that 25% of each corporate prize funding also flows into the general prize pool.

For a lot of these open-source model companies, it's less about the $s (though $200K is nothing to sneeze at), it's the clear recognition that helps their model efforts stand out, gain usage etc.

[NitpickLawyer]

I think the Korean government did have a competition like this, I remember last year we got a bunch of models released at the same time to make the cut for the next stage. The models weren't anything to write home about, IMO.

Having it with clear hw requirements tiers is a nice differentiator. The only issue is that the benchmarks would 100% need to be closed, no other way around it. And then you have the issue of creating and curating good evals for every "stage" of the project. That's a hard task even for "honest" lab-internal evals. And you'd have to publish those evals after each round (for trust purposes), and start over for the next round. Doable, but it would cost a lot (probably more than the prize pools) and you'd have to keep doing this.

[steve_j_choi]

It was a competition organized by the Korean government but the directive wasn't for the same cause as the writer. It was more for constructing Sovereign AI for the country. Also, all models except Exaone had some weights copied from Chinese models, and from the corporate point of view, developing from-scratch model is not cheap despite the financial support from the government.

Yes, I hope the open model communities will someday be able to run current frontier models which will be able to handle autonomous tasks and the hardware to run it will be served at consumer level; however, like how recycling isn't profitable, no companies will fully commit to the movement. Don't get it twisted, I don't have a solution but maybe a global scale movement to liberate knowledge-library could suffice.

[BrtByte]

Yeah, I suspect the $200K check would be the cheapest line item in the whole thing

[rao-v]

Pretty sure the labs subsidized the well known benchmarks w free tokens for evals

[patches11]

This seems like a good idea but also just fun. I can’t train a frontier model but maybe I could compete in the 16 GB tier. I would suspect there are a ton of optimizations out there for the taking that aren’t being considered because frontier models are way above these weight classes

[BrtByte]

I would just add a reproducibility requirement and avoid keeping the exact same benchmarks for too long

[PunchyHamster]

Not sure that would even cover power for training