[gritzko]

LLM-is-a-compiler is indeed a simplistic approach. I wrote a rebuttal to the yesterday's Cursor post, may reuse it here https://replicated.live/blog/follow-up

The idea that a 835-page spec "just exists" and we run an LLM to implement it is completely flawed. Specs do not appear out of nowhere, they co-evolve with the code. If you have the code, why do you want to generate it again?

Good software is made as a product of numerous feedback loops and LLMs let you operate those loops faster. They do not supplement the entire process though.

In the end, a good product is a barrel of distilled feedback.

[atomicnumber3]

Something I have always (even pre-LLMs) found funny to think about is - all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere. So, suppose you want a specific program.

1. Some huge % of those possible programs are obviously not the one you want (most don't even compile).

2. Remaining programs might look similar to the one you want, but are buggy enough to be completely unusable, unreadable to the point of intractability, and so on.

3. Remaining programs look substantially correct but upon using for > a few mins you note major bugs that make it still not correct enough.

4. Remaining programs look substantially correct and seem to generally do your will but contain a long-tail showstopper subtle bug that corrupts saved data, or makes all the output subtly incorrect, etc.

5. Remaining programs might be useful, even if they're mildly annoying.

6. This process can probably continue for several thousand iterations until you finally find "it." The program you wanted. Or... one of them. There's probably still 10k+ candidate programs left at this stage.

Our job has always been to get to 5 and aspire to 6. Indeed most of software development is just doing 5->6 in a loop.

I think LLMs help us get to somewhere between 3 and 5 faster than we used to. And a big problem with them is that programs in 3 vs 5 all already look substantially correct and there's no way to know if you're getting 3 or 5.

Generally, the above is not how it _felt_ to write software pre-LLMs, it was just a cute way to imagine what you're doing. Now it's weirdly apropos.

[dpoloncsak]

I once saw a YouTube video where two beginners were learning the basics programming, and upon opening a cmd prompt, one of them said "It's insane if you just type the right words into this box, you can steal someone's 401K in Arkansas" (Or something along those lines). It's always stuck with me that technically that's pretty correct. It's probably not how you'd go about it, but with the correct magic words in the right order it would work

[atn34]

One of my professors used an example like this to encourage you to pause if you ever find yourself doing something like "maybe if I just flip this < to > the tests will pass". The space of incorrect programs (even ones that pass your tests) is enormous compared to the space of correct programs.

If you like thinking about this and haven't seen this before, check out the busy beaver problem[1].

[1]: https://en.wikipedia.org/wiki/Busy_beaver

[JKCalhoun]

I remember the Computer Recreations column in Scientific American on busy beavers back in 1984. I was too young then to really appreciate it. (I really miss that column—and the Amateur Scientist.)

I found it here (page 19) https://wkbpic.com/wkbx/SA/1984/1984-08-01.pdf

[Xmd5a]

flipping > to >= works quite often though!

[lilbigdoot]

I was about to say a lot of algorithms or invariant bugs for me are slight mistakes at an edge of something

[oblio]

"There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."

[lilbigdoot]

I've come to consider memory management the third

[chubot]

all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere

This seems either mathematically impossible or vacuous, depending on what you mean ...

[maweki]

I think the intent is clear. We could just enumerate all strings in size and lexicographical order. Then the task is only to filter, which strings are programs we desire.

It's the same sentiment as "the typing itself is not the issue, it's the order of the button presses that's difficult."

Of course, by Rice's theorem, all interesting questions are undecidable anyway...

[adrianN]

If we restrict ourselves to programs that can be stored inside the physical universe, suddenly everything switches from undecidable to O(1) :)

[all2]

It's the same mindset some artists have when considering raw stone for a statue: there's a Greek god in there somewhere, you just have to find him.

It's a way of thinking, not some deep metaphysical mystery. Unless it is actually a deep metaphysical mystery, but I'd rather not go there.

[agentultra]

You should check out Borges’, The Library of Babel.

Basically if you have an infinite string of characters in some language then you have every possible book, tweet, script, scribble, and scrap of everything ever written. You just have to find it.

Thats already a large universe to cover. Programs? Much bigger set.

[dfee]

the infinite string must have more refined properties than infinite length. an infinite string of a repeating char doesn't encode much of interest.

[r2_pilot]

I believe it's more that the library contains the finite set of all books which have a (unique) permutation of the language's characters to size N. The challenge to finding the cure to cancer in such a collection of sufficiently large N is indeed finding which book it is in.

[adrianN]

> all code possible to run on a computer already exists

You have to be careful; that line of thinking might get you trapped in L-space. You’d need a very skilled librarian and possibly a bag of peanuts to get back out.

[MattPerry]

Oook

[lilbigdoot]

I think this is a great abstraction of the process I've never considered in those terms and I like it. It explains a bit of how software development differs from some other building processes where the desired outcome is being discovered as you figure out how to make something that produces that outcome

[QuercusMax]

There are plenty of processes that have this property, though. For example: setting up a home, workspace, kitchen, factory, office, etc. to fit a particular purpose. As you discover what works well in a particular space, you will refactor your workflows, seating organization, storage systems, etc to fit what you're doing. And it may even have multiple valid configurations - for example seasonal variations like outdoor vs indoor areas.

[Merad]

> It's some permutation of all the bits of available memory. It's in there somewhere.

True, but the number of possible permutations is so large it's incomprehensible. Let's limit ourselves to say a 50 MiB program, meaning the size of the compiled binary. If I'm remembering the math correctly, that amount of memory can hold 2^(2^20 * 50 * 8) different states. That's 2^419430400 or roughly 10^126000000. For comparison, the heat death of the universe is predicted to occur in about 10^100 years, or 10^107 seconds. You'd need to be able to look at more than a million programs per second to examine them all before the end of the universe.

[oblio]

Reality is a fractal (Mandelbrot?) and this is where LLMs will help science. They will help us explore the fuzzy edges faster. LLMs probably won't find new "leaves", but they should be able to cover a lot of useful but "boring" fuzz.

[zeven7]

I've thought similarly about music before. Everything's there in white noise. To get music you just have to selectively remove parts of it.

[Xmd5a]

yes and if you want to code a little swimmer that explores program-space, you'll find there too!

[LeifCarrotson]

In my experience, a spec is often public (or available for purchase by the public from a standards committee for a few hundred dollars, close enough in business terms), but the source code for your competitor's implementations is not available for any price. The first barrier to entry to become a competitor in many industries is to implement an 835 page (note that those are rookie numbers) spec, building software that is compatible with existing data formats or legal frameworks or network protocols or hardware or whatever.

This is another example of the many invisible silos that developers end up in:

1. Some are developing enterprise software for use by their own employer, the spec only exists because it's useful to help the dev team manage the project. In this case, it's totally reasonable to expect the spec to co-evolve with the code. If the release gets pushed out to Q3 or the target has to be adjusted due to unforseen implementation difficulties, so be it.

2. Some are developing contract software for their customers. The spec exists as a communication tool and requirements that allows the developers to know what to build and to inform the customer that what they've built has reached a certain phase of the billing process (including completion). You're on a clock, and if you're late or you don't do what you said you'd do then your customer isn't going to pay you.

3. Some are developing software for abstract customers - either B2B or B2C. Often, the specs here are regulatory or industry-standard, implementing the spec is the function of the business. You're releasing updates and new products to make year-end sales targets or snipe announcements by your competition, and advertising compatibility with various standards is a crucial bullet point in the sales presentation.

Bug reports, feature requests, and especially deadlines have different meanings in each context. The way AI development agents can be used in each silo also varies enormously!

[kloud]

As a counter argument for the compiler perspective: what Cursor did is basically transpiling a project to another language, which is a form of compilation.

If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.

[kimjune01]

I agree with iteration, but that's an orthogonal argument to how effective it is at turning prose to code. You are asking, LLMs are capable of translating a 835-page spec, but where does the spec come from? Follow the stream up from the built software product:

Binary < IR < Code < Spec < Prompt (aka goals)

For LLMs to faithfully fulfill the role of translating spec/prompt to code, it needs enough information about the problem/motivation and the constraints around it. Often, that's done by research or by importing proven modules. The human role left in this compilation chain is a precise-enough expression of the motivation, upstream of the built artifacts.

Edit For example, SWEbench Pro was supposed to be a measure for how good LLMs are at compiling prose into code. At least 15% of those tasks were underdetermined by ambiguous prose in the spec. Of the issues that are specific enough, Claude and Codex indeed do compile to functional code with self-written tests. Source: https://github.com/kimjune01/swebench-pro-audit

[wrs]

Indeed, this is a much more realistic and useful framing of what to do with the current capabilities of LLMs. They work so fast, at such a high level, that you can speedrun the co-evolution process and end up with a spec and code that works. You can actually use second-system (fourth? fifth? I lost track) syndrome to your advantage because it’s so cheap to throw things away. And they don’t just write the system, they also write the test harness, the benchmarks, the failure simulation, the statistics…all the stuff that seems like “overhead” but lets you drive that evolution with data.

I’ve kind of drifted into this mode a bit at a time over the last year, but hadn’t stepped back to make such a coherent explanation of it.

[layer8]

The source code always was the result of that distillation, or at least a crucial part of the result. I’m unconvinced that natural-language specs can replace that role. Even long before LLMs, when writing specs I often resort to some form of formal language to precisely nail down certain parts of the specification. And some parts are always left open, where I know I’ll get to them when writing the code, because they are more effectively solved “in the code” than waterfall-like up front.

[reinitctxoffset]

Well, since we started training them on RLHF-style rating of the last generation they've totally collapsed back onto predictable suffix generators, which is why there are desperate, low-AUC BERT-inspired clasufiers bolted to them now.

They have like, Chomsky grammars now. And you can't fix it, it's structural to the process. You have to rewind almost to the pre-train.

[imagent]

A zen master asked a junior engineer to go get him a rock. The junior engineer asked what kind of rock he should get and the master replied by knocking him in the head with his cane. The junior thought for a minute, then asked, "would you like a round rock or a flat rock?" Again, the zen master rapped him on the head. The junior thought for a moment, then went to the creek and fetched a rock at random. He brought it to the zen master who inspected it and then said, "No, I want a flat piece of flint for starting a fire." The junior engineer was then enlightened.

Stakeholders will only give you requirements after they see the prototype. Thus, the spec and the prototype to elicit it are two sides to the same coin. Once the spec if correct and the prototype is starting to show its age, throw away the prototype and implement the spec with a fresh start.

[skydhash]

> Stakeholders will only give you requirements after they see the prototype.

That’s very much incorrect and sounds like someone who love building more than communicating. Even without a prototype, users and other consumers will be able to explain their problems. It may not use the same metaphor map that you’re used to. But it’s very much a description of the problem space.

Once you design a prototype, what you will get is feedback about the solution you’ve designed, not the raw problem. If you’ve not listened well in the first place you may be well off the mark, and have to work harder to correct things.

Even if your story, the junior would do way less work by asking the master what he intends to do with the piece of work instead of focusing on the rock itself.

[kimjune01]

whether the hypothesis originates at the users or in your guess for the users is irrelevant in that they're both hypotheses with no certain outcome until trial.

[skydhash]

They are. But engineering is about minimizing costs in solving a problem while ensuring requirements are met. Not worrying about costs is not engineering. It's either a research project (where you want to know if something is possible) or playing around.

[imagent]

The cost of prototyping has dropped so much due to LLMs and coding agents that it basically always makes sense to prototype something to elicit feedback, just as part of the requirements gathering process. E.g. if you can spend $50 to understand something that will save you two weeks of effort, wouldn't you do that? That's the sort of dynamic I'm talking about.

[skydhash]

The cost of prototyping has always been very low. I can quickly sketch a UX flow using balsamiq. Or copy-paste paste code from docs and examples to show a rough working model with. If you can extract the essence of the problem you can quickly realize cheap ways to demonstrate a solution, and not spend two weeks trying to code it.

If there’s something that is wrong with using LLMs to prototype, it is that those prototypes are always too complex. Instead of decomposing a problem and testing unrelated concerns apart from each other, it’s often a mix of everything that the LLM user had been thinking about.

Often there’s no rapid iteration where you are comparing several approaches. Instead it became tunnel vision and confirmation bias where if it’s working, that means it’s the solution.