[bopbopbop7]

I'll believe it when I start seeing examples of good and useful software being created with LLMs or some increase in software quality. So far it's just AI doom posting, hype bloggers that haven't shipped anything, anecdotes without evidence, increase in CVEs, increase in outages, and degraded software quality.

[Mtinie]

It would be helpful if you could define “useful” in this context.

I’ve built a number of team-specific tools with LLM agents over the past year that save each of us tens of hours a month.

They don’t scale beyond me and my six coworkers, and were never designed to, but they solve challenges we’d previously worked through manually and allow us to focus on more important tasks.

The code may be non-optimal and won’t become the base of a new startup. I’m fine with that.

It’s also worth noting that your evidence list (increased CVEs, outages, degraded quality) is exclusively about what happens when LLMs are dropped into existing development workflows. That’s a real concern, but it’s a different conversation from whether LLMs create useful software.

My tools weren’t degraded versions of something an engineer would have built better. They’re net-new capability that was never going to get engineering resources in the first place. The counterfactual in my case isn’t “worse software”—it’s “no software.“

[bopbopbop7]

It really shouldn't be this hard to just provide one piece of evidence. Is anecdotes of toy internal greenfield projects that could probably be built with a drag and drop no-code editor really the best from this LLM revolution?

[Mtinie]

What is your bar for “useful”? Let’s start there and we’ll see what evidence can be offered.

User count? Domain? Scope of development?

You have something in mind, obviously.

[bopbopbop7]

If you're asking me to define a very clear bar, it's obvious nothing cleared it.

Anything that proves that LLMs increase software quality. Any software built with an LLM that is actually in production, survives maintenance, doesn't have 100 CVEs, that people actually use.

[zeroonetwothree]

At my work ~90% of code is now LLM generated. It's not "new" software in the sense that you're describing but it's new features, bug fixes, and so on to the software that we all work on. (Although we are working on something that we can hopefully open source later this year that is close to 100% LLM generated, and I can say, as someone that has been reviewing most of the code, is quite high quality)

[bgar]

Well, on the surface it may seem like there’s nothing being created of value, but I can assure you every company from seed stage to unicorns are heavily using claude code, cursor, and the like to produce software. At this point, most software you touch has been modified and enhanced with the use of LLMs. The difference in pace of shipping with and without AI assistance is staggering.

[bopbopbop7]

> every company from seed stage to unicorns are heavily using claude code, cursor, and the like to produce software

> The difference in pace of shipping with and without AI assistance is staggering.

Lets back up these statements with some evidence, something quantitative, not just what pre-IPO AI marketing blog posts are telling you.

[bgar]

Why quantitative? I have friends at most major tech companies and I work at a startup now. You shouldn’t write by hand what can be prompted. Doesn’t mean the hard parts shouldn’t be done with the same care as when everything was handwritten, but a lot of minutiae are irrelevant now.

[bopbopbop7]

Because anything not quantitative is either “trust me bro” or AI marketing. Some of us are engineers, so we want to see actual numbers and not vibes.

And there are studies on this subject, like the MITRE study that shows that development speed decreases with LLMs while developers think it increases speed.

[zeroonetwothree]

It's not clear what evidence you expect to see? Every major tech company is using AI for a significant % of their code. Ask anyone that works there and you will have all the evidence you need.

[bopbopbop7]

> Every major tech company is using AI for a significant % of their code

It shows, increased outages, increased vulnerabilities, windows failing to boot, windows task bar is still react native and barely works. And I have spoken to engineers at FANG companies, they are forced to use LLMs, managers are literally tracking metrics. So where is all this amazing new software and software quality or increased productivity from them?

[stoneforger]

You are measuring differently..they measure in how much stuff they ship. They don't measure if it's going to break , if it is not very maintainable, or how much it will cost to keep using the LLM I. The future. Remember its an extraction grift: buy now, pay later. Preferably after you have made the model an intrinsic part of the process. Oh snap, now we definitely need to bailout LLMs cause noone knows how this stuff works. Please help. Useful idiots all around. Classic case of not using their brains the way they evolved for.

[cmiles74]

Like the new features in Windows 11? They’ve just anointed a “software quality czar” and I suspect this is not coincidence.