[crabbone]

A lot of programmers naively believe that tests prove that the program works... Even though this has been repeated over and over again: tests can only prove that the program isn't broken in some specific way.

When there's a person in the loop, you know that the program was written intentionally. It can still be wrong, but if you were to ask the author why they wrote this or another part of the program, they'd have an explanation ready. When you interact with LLM, it can generate an explanation, but, fundamentally, it doesn't work in the same way the flesh-and-blood programmer does. It doesn't really have an explanation. LLM can be right 99 times out of 100, where a human undertaking the same task might be right only 90 times out of 100, but the inability to find that 1 wrong case is scary.

LLMs and live programmers make mistakes in different ways. You, the tester, can re-trace the thought process of a live programmer and detect errors where your outcomes don't match the outcomes produced by another programmer. You, however, cannot have the same though process as an LLM... that's physically impossible. So, once it's wrong, you are on a wild goose hunt after the error.

[mohsen1]

I personally think at some complexity level almost nobody who wrote the code can tell you what's going on. E.g. TypeScript's `checker.ts`

[crabbone]

I think you are putting the cart before the horse: the code was written intentionally, i.e. someone first had a thought, and then fleshed it out in the code (and maybe had forgotten all about it the minute later).

When LLM generates code, it also has a "thought process" of sorts. But it's very different from the one humans use. An LLM generating code may pretend to have a thought process similar to humans (when asked to elaborate on the reasons the code is this way or the other), but, fundamentally, it's going to be a lie, because the real reasons this LLM created the code in a particular way is very different, and is very hard to grasp for a human, even if they are familiar with the problem (there were some attempts at explaining LLMs "thinking" that would end up looking like heatmaps and other weird things that are useful for debugging them).

Imagine that instead of a stack trace, the program you are debugging spits out only the memory dump with values in registers etc. (the gibberish-looking part of the coredump file). You'd be in a similar situation receiving the "true" explanation from an LLM generating your code.