[Twirrim]

I've a bunch of technical, but non-engineering types around me, and a few of us engineers keep banging the gong on the fact that they can't trust the output of an LLM. That the best way to leverage AI is to get it to write the code in whatever language they prefer, so they get a simple and repeatable tool out the other side. In many regards it's a liberating tool when used like that. I've got TPMs that are really able to use it as a force multiplier for themselves, building small tools that help them, without having to tie up engineers to produce it.

In numerous cases, though, there are folks asking it to go interrogate some stuff they've set up MCPs for, and produce reports from it. If you do that it will give you a different answer every time, even from exactly the same input (because that's how LLMs work) and you just can't guarantee that any of them are accurate. It's a probabilistic layer, and the reports you need to generate need to be deterministic.

The problem is we're so accustomed to the deterministic nature of the large majority of the software we work with. The output is plausible, too, which only exasperates the problem. Folks just assume it's correct.