[Traubenfuchs]

> Completely different results as LLM is non-deterministic.

You'd need to produce this like 20 times by each model and then do 2x20x20 cross comparisons by both models and ultimately distill the 2x20x20 comparison results into two reports of how they differ.

In this non deterministic computing future, everything else is voodoo, feelings and "vibes".

[jamesrcole]

I would expect a model's result each time to be of a similar quality to the other times. There's something wrong if it does a way better or worse job, at the same problem, sometimes. It's possible, but I haven't heard anyone saying that they do.

[jdiff]

No, that's not how LLMs work. It's all probabilities, and that issue has only deepened with providers silently falling back to worse models if they suspect you might be distilling their models. If an LLM rolls a bad token that can tip the whole balance of the response into utter nonsense.

People use LLMs to do vulnerability scanning by throwing them repeatedly at a codebase. Depending on the run they return with nothing, with a false positive, with a true vulnerability. These are very different destinations when faced with the same problem, sometimes.

Since GPT2, people have been throwing a ton of crap at the wall just to pick out one nugget that's uncharacteristically more solid than the others. Honestly? It's not just possible—it's core to how they operate. And it always has been.

[fouc]

probably depends a lot on the temperature setting, lower it is, the more similar the quality perhaps. the higher it is the more variance.