[codegladiator]

for me reliable = deterministically repeatable. if a llm has been able to do a task successfully and i ask it to do the same task again, i want the reliability that i get the same outcome (be if failure or success)

like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.

[samatman]

This seems too strong to me. I agree with reliable as "repeatable", it's the determinism which seems excess to purpose.

If I have a great meal every time I go to a restaurant, that chef is reliably good. I do not require that the tomatoes on the salad be of identical number or placement, or cut along the precise same bias. Nor would I count the sesame seeds on the bun.

Are LLMs reliable by that standard? Sometimes yes, for some tasks. The envelope of tasks for which they're reliable is continuing to grow.