[danielmarkbruce]

You don't just try every model that comes along... and for the ones you do, you have eval pipelines set up. It's engineering, you have to test stuff works.

Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.

[overgard]

Well, first off, I don't need to test that stuff works, the labs need to do that! I can afford to wait a couple days to get an impression on how people like a model before I switch. I don't want to be a beta tester.

But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are useless and gamed. There's not a lot of transparency in how they're trained, so there's no way for me to go like "Well, I care about how much graphics programming it's been trained on, and I can see X ingested Y amount of training data on that.." It's basically a stab in the dark of very frothy people saying "X CHANGES EVERYTHING!" with no particular details other than a bunch of guys on youtube being like "I gave it a really lame prompt and it made me this game that absolutely nobody would ever want to play!"

[danielmarkbruce]

You are completely wrong and will never make a good product/service with that attitude. You need to test the models work for your use case.

Knowing how much of type X data a model was trained on isn't going to help you one bit. Various phases of training can effectively wipe out the training from earlier steps if not done well. All that matters is does the model do what you want it to do. In practice almost everything in AI/ML is empirical.

[overgard]

Mate, I have 20+ years of experience writing software, I'm not really worried. Frankly if I never used a model ever again I would still write software just fine. These things are conveniences to me, not load bearing.