[jcgrillo]

I'm curious whether you have any insights into why high quality LLM-assisted (or enabled?) projects seem so relatively rare? Instead it seems like the preponderant contribution is a deluge of low quality slop.

You didn't share yours and Adam's prompts in the post, so I'm left wondering how much of the success of this project is attributable to your collective ability and experience (both with this particular project and software in general) vs the capability of the model and harness itself? On that note, do you anticipate releasing LLMs at Oxide[1] (linked from RFD 0576)?

Personally I find credible success stories like yours interesting, if a little jarring. If they were commonplace, shouldn't software be generally getting a lot better?

[1] https://github.com/oxidecomputer/meta/tree/master/engineerin...

[rincebrain]

There exist a great many people in the world who think that the only important thing is having a good enough idea, and everything else is almost valueless by comparison. You've probably met them, people who say things like "I just need someone to code it, can you sign this NDA, what do you mean you want to be paid, it's just coding?"

They exist in other formats too - blogs in the vein of "for exposure" cover the same premise, mostly.

Vibe coding has allowed them all to try and show everyone how right they were.

[jcgrillo]

Right, but most of them don't already work as software engineers, at least it hasn't been my experience of my colleagues. However, companies that are aggressively adopting AI coding tools aren't (at least nobody has shown it) getting better by any metric. So, what gives? Why, generally, isn't this kind of success story common?

[rincebrain]

Because most tasks aren't "refresh this code from 10-20 years ago"?

If you needed an old piece of code at $WORK, you probably already paid the tax of refreshing it or replacing it.

This sort of task is similar in nature to something like "I have a 25yo unmaintained Linux driver, let's refresh it for modern Linux" - a great demonstration of the efficacy of these tools if you have the right-shaped task, but not a task that comes up repeatedly in most people's days.

[jcgrillo]

That's a good point, this does seem qualitatively different. Like dialect translation. In that case the specification is really precise, it's just the old code. Building something new or adding functionality to existing software the spec is almost guaranteed to be more vague.

EDIT: this specificity seems important for language models but the harder I think about it the less sure I am that it's the right intuition..

[rincebrain]

My intuition is that what makes them well suited is that the transformations on the input desired are well-defined and frequent tasks - e.g. any other software that migrated from, say, SDL1 to SDL2, or had to move from gcc 3 to 4, or Sun cc to gcc, had to have these transformations in their source history.

IOW, "there is probably very little stopping you except time from having written Coccinelle patches to mechanically do most of these transformations".

[RugnirViking]

for the same reason that there is so much bad writing online? you'd think that making it much easier to make large projects would mean a great many more people would make large projects - this will inevitably reduce quality.

The number of people with the time and dilligence, the people who previously would have been making them previously, is much fewer. They hide themselves away to find likeminded people anyway, but now its even harder to find them amongst all the deafening slop.