[davepeck]

I see less "painting as a luddite" in response to statements like this, and more... surprise. Mild skepticism, perhaps!

Your experience diverges from that of other experienced devs who have used the same tools, on probably similar projects, and reached different conclusions.

That includes me, for what it's worth. I'm a graybeard whose current work is primarily cloud data pipelines that end in fullstack web. Like most devs who have fully embraced LLMs, I don't think they are a magical panacea. But I've found many cases where they're unquestionably an accelerant -- more than enough to justify the cost.

I don't mean to say your conclusions are wrong. There seems to be a bimodal distribution amongst devs. I suspect there's something about _how_ these tools are used by each dev, and in the specific circumstances/codebases/social contexts, that leads to quite different outcomes. I would love to read a better investigation of this.

[efnx]

I think it also depends on _what_ the domain is, and also to a certain degree the tools / stack you use. LLMs aren’t coherent or correct when working on novel problems, novel domains or using novel tools.

They’re great for doing something that has been done before, but their hallucinations are wildly incorrect when novelty is at play - and I’ll add they’re always very authoritative! I’m glad my languages of choice have a compiler!

[jpc0]

My recent example for where its helpful.

Pretty nice at autocomplete. Like writing json tags in go structs. Can just autocomplete that's stuff for me no problem, it saved me seconds per line, seconds I tell you.

It's stupid as well... Autofilled a function, looks correct. Reread it 10 minutes later and well... Minor mistake that would have caused a crash at runtime. It looked correct but in reality it just didn't have enough context ( the context is in an external doc on my second screen ... ) and there was no way it would ever have guessed the correct code.

It took me longer to figure out why the code looked wrong than if I had just typed it myself.

Did it speed up my workflow on code I could have given a junior to write? Not really, but some parts were quicker while other were slower.

And imagine if that code bad crashed in production next week instead of right now while the whole context is still in my head. Maybe that would be hours of debugging time...

Maybe as parent said, for a domain where you are braking new ground, it can generate some interesting ideas you wouldn't have thought about. Like a stupid pair that can get you out if a local manima but in general doesn't help much it can be a significant help.

But then again you could do what has been done for decades and speak to another human about the problem, at least they may have signed the same NDA as you...

[davepeck]

Yeah, absolutely.

LLMs work best for code when both (a) there's sufficient relevant training data aka we're not doing something particularly novel and (b) there's sufficient context from the current codebase to pick up expected patterns, the peculiarities of the domain models, etc.

Drop (a) and get comical hallucinations; drop (b) and quickly find that LLMs are deeply mediocre at top-level architectural and framework/library choices.

Perhaps there's also a (c) related to precision. You can write code to issue a SQL query and return JSON from an API endpoint in multiple just-fine ways. Misplace a pthread_mutex_lock, however, and you're in trouble. I certainly don't trust LLMs to get things like this right!

(It's worth mentioning that "novelty" is a tough concept in the context of LLM training data. For instance, maybe nobody has implemented a font rasterizer in Rust before, but plenty of people have written font rasterizers and plenty of others have written Rust; LLMs seem quite good at synthesizing the two.)