[jimminyx]

> Testing and mocking is a huge challenge when developing LLM driven systems that aren’t deterministic. Even relatively simple flows are extremely hard to reproduce.

This is by far the most frustrating part of building with LLMs. Is there any good solution out there for any framework?

[th0ma5]

This in my opinion is the fundamental problem where because LLMs work on a computer there is an assumption of correctness in surprising ways. In reality, when you mix a deterministic system and a probabilistic system you always get a probabilistic system.