[overgard]

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.

I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.

[drusepth]

One of the best features I built into our game studio's AI router (IMO) was to take a random sampling of generations and have them generate across ALL models (that we support, at least) so we, the developers, can browse outputs and get a sense each model's output. Seeing them all side by side for generations you're already familiar with makes it feel like significantly less cognitive load.

The better I get at recognizing what each model is good/bad at, the more I'm glad we're taking the time to choose specific models for specific prompts -- and the more I wouldn't trust a generic router to efficiently route for me.

[fennecbutt]

This is a problem we're looking at atm. We want to start running evaluations as we're having problems with consistency across releases even just for individual agents.

[cyanregiment]

There are "Hi" and "Lo" variants of the different models.

Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job.

Find the same with image, video, vision, and so-on.

Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR.

I don't find a ton of competitive difference amongst the "good ones" especially not reflected in the pricing and marketing. IMO there are pretty much just "non-AI" algorithms, "basic AI" that are getting more interesting on-device for some use cases, and "good AI" or what we'd call "frontier models" - which again, not seeing a ton of differentiation on that layer.

[overgard]

I concur that there isn't that much difference between the good ones. I'm not really trying to push the limits of what they can do or anything, but I'd say on average I give it tasks about the size of something that would take me a half day to do by hand, and honestly, other than minor quirks in coding style and word choices, they've all basically been "good enough" for what I want. When I tried fable it just wasn't for me because it was basically very expensive overkill. I usually stay on the older models as long as I can just because they're predictable and I care a lot more about predictability and cost than being on the super cutting edge or whatever. (Honestly I'd kind of rather the enthusiasts test it first for the various "new model deleted my database" bugs that seem to pop up.

[cyanregiment]

Yeah definitely good enough for work. I think people only get into trouble going “agentic” and trying to automate je ne sais quoi

[LoganDark]

I feel like nearly anything done by a "Lo" is better done myself. I use "Hi" for pushing the limits of my expertise.

[cyanregiment]

A good use case for me is automating invoice notes that are all very similar, but change based on details of client and work. Mistral 7b does it fine, instantly.

UX-wise, I don't predict out "ghost text" like Google has been doing, I just preload the input value with the prediction. You can edit as you want. Saves a ton of time.

[mrkeen]

A programmer is someone who declares "I understand the problem so well that I can teach a computer to solve the problem, even for examples I have never seen before".

If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.

[jurgenburgen]

There’s a heuristic I’ve been using to pick which model to use. The provider is already chosen by my org so what I do is activate the model picker in my harness, select the most expensive model’s latest version and set it on the highest effort setting.

This has worked quite well for me and saves a lot of time.

[danielmarkbruce]

You don't just try every model that comes along... and for the ones you do, you have eval pipelines set up. It's engineering, you have to test stuff works.

Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.

[overgard]

Well, first off, I don't need to test that stuff works, the labs need to do that! I can afford to wait a couple days to get an impression on how people like a model before I switch. I don't want to be a beta tester.

But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are useless and gamed. There's not a lot of transparency in how they're trained, so there's no way for me to go like "Well, I care about how much graphics programming it's been trained on, and I can see X ingested Y amount of training data on that.." It's basically a stab in the dark of very frothy people saying "X CHANGES EVERYTHING!" with no particular details other than a bunch of guys on youtube being like "I gave it a really lame prompt and it made me this game that absolutely nobody would ever want to play!"

[danielmarkbruce]

You are completely wrong and will never make a good product/service with that attitude. You need to test the models work for your use case.

Knowing how much of type X data a model was trained on isn't going to help you one bit. Various phases of training can effectively wipe out the training from earlier steps if not done well. All that matters is does the model do what you want it to do. In practice almost everything in AI/ML is empirical.

[overgard]

Mate, I have 20+ years of experience writing software, I'm not really worried. Frankly if I never used a model ever again I would still write software just fine. These things are conveniences to me, not load bearing.

[brunaxLorax]

It's hard to keep track of all those launches, I agree. I would separate 2 usages of LLMs: - LLM is integrated into a workflow/app for a classification or whatever job: scoring outputs (evals, KPIs) and choosing a fit between quality and cost is enough - LLM is used as a tool you interact with (like coding or research): here is where you have to know your brushes. As humans we get used very quickly by the model personality, the format of output, the way it "thinks" and interacts.

[daeken]

I just ask our fleet maintenance agent to configure the variants of the new model and then run the bakeoff. In a few hours, I get a doc containing exactly how it performed on my real tasks, against the other models we've benchmarked.

We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.

Good enough to have guided multiple model swaps and measured increases in productivity.

[mountainriver]

You do still need to know though, different prompts have different affects on them.

[zackify]

I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.

[bobmcnamara]

Found the drywaller

[cyanregiment]

My landlord just muds over cob webs and open space. They're over 100 years old so they legally count as part of the building's foundation.

So is he, and so does he. He looks like the Tales From The Crypt guy I have no idea how he's still doing manual labor.

[stingraycharles]

Just don’t switch models every week and you’ll be fine. Especially in production.

[overgard]

I definitely wouldn't use a model for anything in production!

[inigyou]

I'm not surprised that people who use LLMs all day for everything get a feel for the different ones.

[TZubiri]

I'm all for dropping priority on vendor neutrality and tightly coupling with vendors.

But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.

I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.

[iLoveOncall]

Yeah I'm growing increasingly tired of those stupid metaphors about AI.

LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic.

I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.

[paradoxyl]

AIs are like (insert clickbait keyword). You just have to (insert clickbait call to action).