for me reliable = deterministically repeatable. if a llm has been able to do a task successfully and i ask it to do the same task again, i want the reliability that i get the same outcome (be if failure or success)
like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.
This seems too strong to me. I agree with reliable as "repeatable", it's the determinism which seems excess to purpose.
If I have a great meal every time I go to a restaurant, that chef is reliably good. I do not require that the tomatoes on the salad be of identical number or placement, or cut along the precise same bias. Nor would I count the sesame seeds on the bun.
Are LLMs reliable by that standard? Sometimes yes, for some tasks. The envelope of tasks for which they're reliable is continuing to grow.
like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.