[emmender2]

Joel Spolsky had a great article on leaky abstractions.

LLMs for code are leaky abstractions. They work many-a-time. But when they break, good luck fixing it.

yann-lecunn also put it well. If something works 95% of the time, and you compose it 10 times, it only works 59% of the time.

In the real world of software engineering, we cannot build on something that works 95% of the time reliably. And LLM apologists will immediately say that code written by humans has bugs too. Of course it does.

[mrbungie]

Wouldn't this apply to any kind of numerical or logical model (i.e. a model learned over preexisting data) used in a software solution?

[emmender2]

indeed, and that goes into the heart of it.

ie, things we construct by the computer are deterministic. the turing machine (and other equivalent models like the lambda calculus etc) being the canonical machine that models our computations. Arguably, all human knowledge is symbolic and determistic - even though it may model probabilistic phenomena.