[DanielHB]

One of the main problems I face is that business knowledge is not explicitly written down and formulated. No matter how much you describe something in human text format those are not formal rules about how something should work. If you can't inspect the output of the LLM you do not have full understanding of the business.

Most "classic" computer systems (code that runs on a CPU) are not strictly formal[1], but they are orders of magnitude more formal than human language.

[1] things like distributed systems, memory corruption, performance characteristics, etc can make a system behavior not predictable