[itay-maman]

My first reaction: wow, incredible.

My second reaction: still incredible, but noting that a C compiler is one of the most rigorously specified pieces of software out there. The spec is precise, the expected behavior is well-defined, and test cases are unambiguous.

I'm curious how well this translates to the kind of work most of us do day-to-day where requirements are fuzzy, many edge cases are discovered on the go, and what we want to build is a moving target.

[ndesaulniers]

> C compiler is one of the most rigorously specified pieces of software out there

/me Laughs in "unspecified behavior."

[ori_b]

There's undefined behavior, which is quite well specified. What do you mean by unspecified behavior? Do you have an example?

[ndesaulniers]

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3685.pdf

Read section J.1.

[irishcoffee]

Undefined is absolutely clear in the spec.

Unspecified is whatever you want it to mean. I am also laughing, having never heard "unspecified" before.

[LiamPowell]

Unspecified behaviour is defined in the glossary at the start of the spec and the term "unspecified" appears over a hundred times...

[astrange]

The C spec is certainly not formal or precise.

https://www.ralfj.de/blog/2020/12/14/provenance.html

Another example is that it's unclear from the standard if you can write malloc() in C.

[butterNaN]

Sure but the point OP is making is that it is still more spec'd than most real world problems

[astrange]

You're welcome to try writing a C compiler and standard library doing no research other than reading the spec.

[cryptonector]

> My second reaction:

This is the key: the more you constrain the LLM, the better it will perform. At least that's my experience with Claude. When working with existing code, the better the code to begin with, the better Claude performs, while if the code has issues then Claude can end up spinning its wheels.

[softwaredoug]

Yes I think any codegen with a lot of tests and verification is more about “fitting” to the tests. Like fitting an ML model. It’s model training, not coding.

But a lot of programming we discover correctness as we go, one reason humans don’t completely exit the loop. We need to see and build tests as we go, giving them particular care and attention to ensure they test what matters.

[uywykjdskn]

The agent can obviously do that