[the_real_cher]

This person is taking things too literally.

By power tool we simply mean speeds up work.

And by that definition LLMs are absolutely a power tool of epic proprtions!

If you ever coded before LLMs existed its like night and day.

[inigyou]

Evidence from listening to slopcoders speak says they feel it's night and day. Evidence from observing reality shows a 20% speed increase at best and 50% speed decrease at worst and almost always a significant reduction in quality.

[Jemaclus]

I was also skeptical, but I started measuring quality (I'm using rework as a rough metric) and quantity (PR throughput), and I've found that, on average, LLMs are 60% worse at writing code (meaning they rework at a rate 60% higher than humans) but my team produce PRs about 2.4x faster. So objectively my team's numbers don't match up with yours: our speed is about 2.5x increase _on average_ and a 60% reduction _on average_ in quality. Some basic math suggests that our ROI is something around 50-60% improvement over just humans.

I'm not convinced that throwing more money (i.e. tokens) at the problem would make us any more efficient, but 1.5x improvement ain't nothing.

And just to address the "night and day" bit.. if you gave a construction foreman a machine that made his team 20% more productive, he'd take that all day long and tell everyone it's "night and day." I'm not sure what your threshold is, but +20% isn't nothing either. That foreman would hit more deadlines than not, and get more bonuses than not.

[jdlshore]

Quality is more about maintainability, IMO, which comes down to consistency and clarity of design, sensible abstractions, cohesion, decoupling, etc. I don’t think a rework metric tells you that.

[inigyou]

That actually is incorrect. Quality is about how well something works.

[jdlshore]

There’s two types of quality: external quality, which is how well something works, and internal quality, which is how well something is built.

[smcg]

Twitter is a rather crude place, but it can be succinct. An exchange I saw recently:

A: "I spent more tokens than the entire OpenCode team combined."

B: "Why aren't you more successful?"

[inigyou]

Telling it like it is. I bet Sam got Musk to restrict user B.

[blainehansen]

Original author here. Metaphors become real, and they shape the way people think. I think it's dangerous to compare LLMs to deterministic systems because it misleads people into treating this situation we're in like other kinds of automation, which it absolutely isn't.

Also, if you want me to believe that LLMs have made you that much more productive, please share some links for useful things you've built recently using them. I'm thrilled if actual things can be made, but I haven't seen it yet.