[pron]

> It's 2026 and the notion that you can get detailed enough requirements and specifications that you can one-shot a perfect solution needs to die.

It's 2026 and the idea that even with detailed-enough requirements you can one-shot even a workable (let alone perfect) solution also needs to die. Anthropic failed to build even something as simple as a workable C compiler, not only with a perfect spec (and reference implementations, both of which the model trained on) but even with thousands of tests painstakingly written over many person-years. Today's models are not yet capable enough to build non-trivial production software without close and careful human supervision, even with perfect specs and perfect tests. Without a perfect spec and a perfect human-written test suite the task is even harder. Maybe in 2027.

[ianbutler]

Sorry where are we seeing that it failed? It compiled multiple projects successfully albeit less optimized.

" It lacks the 16-bit x86 compiler that is necessary to boot Linux out of real mode. For this, it calls out to GCC (the x86_32 and x86_64 compilers are its own).

It does not have its own assembler and linker; these are the very last bits that Claude started automating and are still somewhat buggy. The demo video was produced with a GCC assembler and linker.

The compiler successfully builds many projects, but not all. It's not yet a drop-in replacement for a real compiler. The generated code is not very efficient. Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled.

The Rust code quality is reasonable, but is nowhere near the quality of what an expert Rust programmer might produce. "

For faffing about with a multi agent system that seems like a pretty successful experiment to me.

Source: https://www.anthropic.com/engineering/building-c-compiler

Edit: Like I think people don't realize not even 7 months ago it wasn't writing this at all.

[pron]

> where are we seeing that it failed?

Anthropic said the experiment failed to produce a workable C compiler:

- I tried (hard!) to fix several of the above limitations but wasn’t fully successful. New features and bugfixes frequently broke existing functionality.

- The compiler successfully builds many projects, but not all. It's not yet a drop-in replacement for a real compiler.

(source: https://www.anthropic.com/engineering/building-c-compiler)

Software that cannot be evolved is dead software. That in some PR communications they misrepresented their own engineer's report is beside the point.

> It compiled multiple projects successfully albeit less optimized.

150,000x slower (https://github.com/harshavmb/compare-claude-compiler) is not "less optimised". It's unworkable.

> Like I think people don't realize not even 7 months ago it wasn't writing this at all.

There's no doubt that producing a C compiler that isn't workable and is effectively bricked as it cannot be evolved but still compiles some programs is great progress, but it's still a long way off of auonomously building production software. Can today's LLM do amazing things and offer tremendous help in software development? Absolutely. Can they write production software without careful and close human supervision? Not yet. That's not disparagement, just an observation of where we are today.

[tardedmeme]

This evaluation appears to be AI-written itself. It claims a 3x slowdown and a 4x slowdown combine to produce a 158000x slowdown "because there are billions of iterations" - yeah well both versions of the program had the same number of iterations.

Does anyone know how the 158000x slowdown happened? That's quite ridiculous.

[leereeves]

It could be written more clearly but I think when it refers to a 4x and a 3x slowdown, it's actually a 4x slowdown and 3x larger code that causes cache misses, and the impact of those cache misses on runtime is surely much larger than 3x.

> Each individual iteration: ~4x slower (register spilling)

> Cache pressure: ~2-3x additional penalty (instructions don't fit in L1/L2 cache)

> Combined over a billion iterations: 158,000x total slowdown

I think that "2-3x additional penalty" refers to this:

> The 2.78x code bloat means more instruction cache misses, which compounds the register spilling penalty.

Also, the analysis refers elsewhere to other factors that weren't included in this part.

[tardedmeme]

3 times 4 is 12. What's the other 157988x caused by?

[ruszki]

You’re searching for 13166x, and not 157988x. “Times” is multiplication not exponentiation.

[ianbutler]

> Can they write production software without careful and close human supervision? Not yet. That's not disparagement, just an observation of where we are today.

I never claimed they could! I just view this as a successful experiment. I don't think anthropic was making that claim with their experiment either.

It feels reflexive to the moment to argue against that claim, but I tend to operate with a bit more nuance than "all good" or "all bad".

[pron]

The experiment failed to produce a workable C compiler despite 1. the job not being particularly hard, 2. the available specs and tests are of a completely higher class of quality than almost any software, not to mention the availability of other implementations that the model trained on.

You can call that a success (as it did something impresssive even though it failed to produce a workable C compiler) but my point in bringing this up was to show that today's models are not yet able to produce production software without close supervision, even when uncharacteristically good specs and hand-written tests exist.

[ianbutler]

That's great and all, but that's not the point I was making and you're engaging rather uncharitably on it. So when you view it from the perspective of capability increase it's rather impressive. Note the slope of progress which this experiment was to show.

Edit: Maybe uncharitably is too strong, but we're talking past each other.

[auggierose]

pron made this statement:

> It's 2026 and the idea that even with detailed-enough requirements you can one-shot even a workable (let alone perfect) solution also needs to die.

and brought up the failed anthropic experiment as proof of that. Yes, you are talking past each other, but that is not pron's fault. It is your fault.

[KajMagnus]

Saying the model failed to write a competitive C compiler makes more sense.

I don't think they tried to do that though.

> today's models are not yet able to produce production software without close supervision, even when uncharacteristically good specs and hand-written tests exist.

That's a good point anyway

[pron]

> Saying the model failed to write a competitive C compiler makes more sense.

Their compiler fails to compile (well, at least link) some C programs altogether, and in other cases it produces code that is 150,000x slower than a real C compiler with optimisations turned off (interestingly, the model trained on the real compiler's source code). That's not "not competitive" but "cannot be used in the real world". But even more importantly, the compiler cannot be fixed or evolved. It's bricked (at least as far as today's models' capabilities go). For any kind of software, not being able to improve or fix anything or add any new feature means it's effectively dead.

You could not use it in production even if no other C compiler existed.

[areweai]

I think people are concerned about the large discrepancy in concrete claims in your previous comment and subsequent empirical information. You may have seen a headline or skimmed an article and missed some details, not a big deal.

The overall impression given was inaccurate and the implicit claim of a fully working end-to-end generated compiler was inaccurate. The headlines were incomplete in a way that was intentionally misleading. It was an interesting experiment and somewhat impressive but the claims were overblown. It happens.

[josephg]

> Sorry where are we seeing that it failed?

Try it yourself.

I've been using claude to make a project over the last few weeks. Its written ~70k LOC to solve a complex problem. I've found that it can get surprisingly far in a 1-shot, but about 90% of the work I've had it do (measured in time and tokens) is cleaning up the junk it outputs in its first pass. I'm finding my claude sessions have a rhythm like this:

1. Plan and implement some new feature.

2. Perform a code review of what you just did. Fix obvious problems. Flag bugs, issues, poor factoring, messy abstractions, etc. Make a prioritised list of things to fix (then fix them).

3. (Later) fixes:

- Write tests for the code you wrote and fix the bugs you find.

- Run the code through memory leak checks, and fix bugs.

- Do a performance analysis using benchmarks and profiling tools, and make any high priority performance improvements.

- Read the whole program, looking for ways in which the code you've just written could fit in better with the rest of the program. Fix any issues.

- In directory X is the full documentation for the library you're using. Reread it then review the code you wrote. Are there better ways we could make use of the library?

And so on.

Claude's 1-shot output is often usable, but its consistently chock full of problems. Bugs. Memory leaks. Bad factoring. Too many globals. Poor use of surrounding code. And so on. Its able to fix many of these problems itself if you prompt it right. (Though even then the code is often still pretty bad in many ways that seem obvious to me).

At the moment I think I'm spending tokens at about a 1:9 ratio of feature work to polish. Maybe its 1-shot output is good enough quality for you. To me its unacceptable. Maybe a few models down the line. But its not there yet.

[mh-]

The ratio is an interesting way of thinking about it. I wonder how this compares to other SWEs at various levels of experience, replacing tokens for person-hours.

[nvme0n1p1]

Why are you quoting from their marketing blog as if it's a reliable source?

https://github.com/anthropics/claudes-c-compiler/issues/1

> Apparently compiling hello world exactly as the README says to is an unfair expectation of the software.

[dnautics]

Yeah I think people are really underestimating what LLMs can do even without specs.

As an example, I did an exploratory attempt to add custom software over some genuinely awful windows software for a scientific imaging station with a proprietary industrial camera. Five days later Claude and I had figured out how to USB-pcap sample images and it's operationalized and smoothly running for months now. 100% of the code written by Claude, it's all clean (reviewed it myself) pretty much all I did was unstuck it at a few places, "hey based on the file sizes it looks like the images are being sent as a 16-bit format")

For day to day work, I'll often identify a bug, "hey, when I shift click on this graphical component, it's not doing the right thing". I go tell Claude to write a RED (failing) integration test, then make it pass.

Zero lines of code manually written. Only occasionally do I have to intervene and rearchitect. Usually thus involves me writing about ten lines of scaffold code, explaining the architectural concept, and telling it to just go

[pron]

People both underestimate and overestimate what LLMs can do. LLMs have shown very different results when autonomously writing a small program for personal use and autonomously writing production software that needs to be evolved for years.

[YZF]

GCC has only like a billion man hours in it?

Assembler and linker are not part of a compiler. They are separate tools. They are also generally much simpler.

[jyounker]

By "non-workable" I think people mean that it won't compile Hello World.

[SirHumphrey]

Most software is much simpler than a c compiler.

[pron]

A workable C compiler is a ~10-50KLOC program, and a fairly simple one at that (batch, with no concurrency or interaction). That Anthropic's swarm of agents wrote 100KLOC before failing is a symptom of the problem. It's certainly possible that many programs are in the sub 5KLOC range, but it's definitely not "most software". Plus, almost no software has this level of detailed spec, ready-made tests, and a selection of existing implementations of the same spec.

My first thought when reading Anthropic's description of the experiment was that it is unrealistically easy. It's hard to come up with realistic jobs in the 10-50KLOC range that would be this easy for an LLM. That it failed only shows how much further we still have to go.

[rowanG077]

The compiler that claude made went way beyond workable. It could compile the full linux kernel afaik. That is much further even beyond standard C.

[pron]

People who independently tried to use it reported that it is very much not workable:

- "CCC compiled every single C source file in the Linux 6.9 kernel without a single compiler error (0 errors, 96 warnings). This is genuinely impressive for a compiler built entirely by an AI. However, the build failed at the linker stage with ~40,784 undefined reference errors."(https://github.com/harshavmb/compare-claude-compiler)

- Overall it’s an interesting experiment, and shows the current bleeding edge of Claude’s Opus 4.6 model. However the resulting product is also a clear example of the throwaway nature of projects generated almost entirely by AI code agents with little human oversight. The prototype is really impressive, but there is no real path forward for it to be further developed. It can build the Linux kernel [for RISC-V], which is impressive. It can also build other things… if you are lucky, but you really cannot rely on it to work. (https://voxelmanip.se/2026/02/06/trying-out-claudes-c-compil...)

Anthropic themselves said that the codebase was effectively bricked and that their agents could not salvage it.

[rowanG077]

Well then as you say a 10-50KLOC C compiler is workable. Could you show me the C compiler that does manage to compile a modern Linux kernel that is of that size?

[pron]

By "workable" I meant something more modest, say, successfully compiling SQLite, something that CCC fails to do unless you stretch the meaning of success (it seems the problem may have to do with register allocation, but I would say that register allocation is pretty much the only interesting job a C compiler does).

As to the Linux kernel, you're right that a compiler that can successfully compile it is likely to be bigger, but we don't know that CCC is able to do it, either. What we do know is that (when using the gcc linker and assembler) people were able to boot the RISC-V kernel in qemu. That's something for sure, but not enough to call it a successfuly compilation.

[spc476]

TCC did several years ago. It could boot Linux from source in under 10 seconds. It's wasn't that big of a C compiler. It's in the 50,000 lines of code range.

[bigstrat2003]

My dude, that compiler couldn't even do hello world successfully. That isn't workable.

[quantumleaper]

A bit off topic, but see how Anthropic publicity stunts went from "Claude C Compiler" with 100K LOC to the recent Bun Rust rewrite with 1M LOC (10x!) in just 3 months.

I get that it's "novel" creation vs porting, but given that they reported that the C compiler cost them $20k in API costs, the Bun rewrite must be at least $200k, maybe even closer to a million. Pure madness.

[gmueckl]

Asking an LLM tp change programming language of an implementation is completely different from asking it to code from spec. It's orders of magnitude simpler in practice. I converted some 60kloc of Java to C++ and it works. There were some issues where the Java implementation used runtime reflection because that needs creative workarounds and not all of the C++ translations worked on the first try. And that was my first serious attempt at a task with an LLM. I could likely do better now. An important task simplification here is that a well designed codebase can be converted in small pieces and then joined back together. So the total amount of code converted becomes an irrelevant metric.

[pron]

Yes, the task is very different, but also it will be months to a year until we know the results of the bun experiment.

[quantumleaper]

I don't know how it could fail - Bun loses popularity among devs? Is it an objective metric? From what I understand, Node.js remains dominant across the industry as a whole, with Deno and Bun mostly used by startups.

Anthropic can always fire the Opus/Mythos token machine gun on any problem (bugs, features, security) to ensure PR success, and there would be plenty of AI-sphere startups already drinking the kool-aid that would consider the whole vibe-coding thing to Bun's benefit.

[pron]

> Anthropic can always fire the Opus/Mythos token machine gun on any problem (bugs, features, security) to ensure PR success,

Can they, though? They tried and failed to do it in their C compiler experiment. The experimenter wrote: "I tried (hard!) to fix several of the above limitations but wasn’t fully successful. New features and bugfixes frequently broke existing functionality."

[eudamoniac]

It could fail due to maintenance burden. There is a lot of code now that no one wrote.

[t_mahmood]

Are we assuming, all tests pass == software done?

Do Firefox not have tests? Then how was there over 200 CVEs found?

Are we going to be comfortable running a piece of software that has 1M lines, and who knows how many zero-days will be in it.

Yes, sure they are going to use LLM to find the CVE's, and so will the hackers. You need a day or two to fix the security issue, a hacker just need to put it in use.

And good luck debugging a million line code base.

1M LOC == already failed.

[binary0010]

Not really.

I can make a c compiler in a couple weeks just by looking up open source libraries and copying them.

I can't make any software that people will pay me money to use without taking months/years of development, research, expiramentation and iteration.

Just because the original people who invented compilers had to be genius, doesn't mean anyone has to spend much time or thought in copying that work now.

[YZF]

I built a compiler for a simpler language as part of my compilers course in a CS degree. It was a non-trivial exercise well beyond the majority of software applications. What open source libraries did you have in mind and what are you copying?

If you can truly write a C compiler in weeks then kudos to you. How many compilers have you written so far for how many languages?

I work for big tech and I would say a large % of developers are incapable of producing a working C compiler on any reasonable time scale, certainly not weeks, even with looking at open source. I'm sure they can download one and run it. Most developers today don't even know C or assembler. They don't know how to approach the C language spec. The top 5-10% of developers/engineers can do it but even for them it's non-trivial.

[pron]

> It was a non-trivial exercise well beyond the majority of software applications

That depends on how you count. By number of programs that may well be right, but that's not what matters in terms of impact on the industry, as software value roughly corresponds to the number of people working on a particular piece of software (or lines of code, if you wish). By number of people/LOC most software is not in the "simpler than a C compiler" category.

[binary0010]

I'd copy and paste from all the thousands of open source ones, what do you mean?

There are plenty of open source compilers that I can copy and paste whatever I need to. I don't get why you think this would have any level of difficulty?

Of course I couldn't make a brand new compiler that was better than what's out there...

Just like a game engine, I could clone one of the thousands of engines out there pretty easily - making something better or novel would be difficult. Just making a bare bones clone of what already exists by referencing documentation and pre-existing code is relatively easy now.

Yeah, when I made a mediocre 3d game engine 20 years ago, it was brain breaking difficult work. I can make one infinitely better in a micro fraction of the time now because most of the hard stuff is done and can just be looked up now.

Do you not agree?

[YZF]

If you copy and paste an entire compiler you didn't make anything. If you copy pieces from different compilers they won't work together. So I'm not sure how you "make" a compiler with copying and pasting from open source compiler. Are you saying you'll take one file from clang, one from gcc, another another from another compiler?

Sure. You can clone gcc and build it. You can close a game engine and use it.

[thayne]

> It was a non-trivial exercise well beyond the majority of software applications

Maybe if you include every application ever written, including every variation of "hello world", but if you are claiming that most serious production quality software could be written by a CS student who is simultaneously working on other classes, I'm gonna have to disagree with you.

[tardedmeme]

I do think being able to write a compiler is a milestone indicator of your computer science knowledge. Most developers probably don't understand pointers either, because "most developers" are people who did a React bootcamp.

[virgilp]

I wonder how knowledgeable in compilation was the engineer that attempted this. I'm pretty confident that I could produce a decent C compiler in a few weeks (or less), if given Opus 4.7 + unlimited tokens + a good test suite. (and this is not blind unsubstantiated belief in AI, I've recently rewritten a somewhat sophisticated interpreter in a week with AI; and have worked on several C++ compilers in the past, including a GCC port to a custom DSP, so I have a bit of an idea about what this would take).

But yeah, this is not a "one shot" project, none of it is. One shot doesn't work even with humans - after all, this is exactly what killed waterfall as a methodology.

[pron]

> I'm pretty confident that I could produce a decent C compiler in a few weeks (or less), if given Opus 4.7 + unlimited tokens + a good test suite.

Of course. The point is that a full, detailed spec isn't enough (even in the rare situations it does exist, like for a C compiler). At least for the moment, you need expert humans to supervise and direct the agents.

Vibe coders usually also let the agents write the tests, which mean that the only independent human validation of the software is some cursory manual inspection. That also obviously isn't enough to validate software.

> One shot doesn't work even with humans - after all, this is exactly what killed waterfall as a methodology.

You can one-shot a C compiler with humans. LLMs' software development ability is impressive and helpful, but it is not human-level yet, even if at some tasks the agents are better than most human programmers. And while many waterfall projects failed, many succeeded (although perhaps not as efficiently as they could have). So far I don't believe agents have been able to produce any non-trivial production software autonomously.

[zem]

yeah, the key part is that there be a human in the loop, directing and course-correcting the ai while it produces code in reasonably small and well defined stages.