[somenameforme]

I think there's a simple dichotomy based on use cases. For things you're already highly skilled at LLMs can be handy but the overall gains, after all is accounted for, are not so clear. But for things you aren't good at, they're zomg amazing. So for somebody evaluating things based on what they do at work (where they're probably quite competent) or on personal projects well within their own domain, then it's 'hey what's all the hype about?' But if you're doing things outside your domain then it's a revolutionary game-changer.

This also explains why an independent dev can proclaim a 10x productivity boost or whatever, and actually seem to be showing that - while major companies dumping obscene amounts of $$$ on tokens don't seem to be have much to show for it.

[InsideOutSanta]

That's one dimension. Another is how much you care.

For my own projects, little productivity apps for my own needs, I'm easily 20x more productive. I'm building so many more of them, and at a much higher level of polish. But that's simply because I don't care about the code, long-term maintenance, or anything else. I just need them to do a thing, and if they do the thing, I'm good. I used to make an effort calculation before building these things, but now I just tell Claude "hey, build this", and two hours later it's built well enough to be useful.

OTOH, at work, I'm probably about as fast as I was before, maybe even a bit slower. But the quality of my work has increased quite a bit, because I no longer take the shortcuts I used to take to push things out. I'm spending much more time in planning and in code reviews rather than actually typing code.

Either way, the idea that these obvious changes are just people fooling themselves is, at this point, no longer a reasonable position. And the claim that "AI will run out of money and just be turned off due to the huge operational cost" is so implausible that I would feel ashamed of myself if I used this as a straw man for what AI skeptics believe.

[dingaling]

I understand what you're saying about side projects but it just sounds... like buying a landscape wall print from IKEA instead of taking a photo and having it printed on canvas.

My non-work projects are exercises for the brain and for learning, and if instead I could just automagic them into existence they'd feel hollow and almost cheating.

[InsideOutSanta]

Sure, it's cheating if you want to frame it like that. But it's still a huge boost in productivity.

[breuleux]

Is it? Genuine question. Say you didn't do these things at all -- what would that change to your life? Are you gaining productivity in high-impact things, or stuff that might scratch an itch but you don't really need all things considered?

[InsideOutSanta]

> what would that change to your life?

Sure, let me give you an example. I bought an Xteink X4 recently so I always have something to read with me. I have a huge dump of epubs from Humble Bundle and other sources. I used to just copy them to my e-book readers and then never read anything because it's impossible to find.

I had Claude write a Python script that sorts the books into a file system structure according to Language → Genre → Sub-Genre → Author. Now the device is actually useful, because I can easily find whatever book I want to read, and it has resulted in my reading much more.

> Are you gaining productivity in high-impact things, or stuff that might scratch an itch but you don't really need all things considered?

I technically don't need much. You could lock me in a room and give me stale bread and water once a day. So I'm not sure why you're applying that standard to the usefulness of LLMs, and I'm not sure what you consider "high-impact things". I'm doing high-impact things for me, but maybe you disagree.

So let me ask you this instead: did you write these types of comments in the past, when people manually built software? If I manually wrote a Python script to sort my books, would you post a comment questioning whether that changed anything in my life?

A lot of the criticism leveled against using LLMs in software development is just criticism against software development in general. Do we really need all this software? I guess not. Can we objectively measure the productivity of software developers? Not really.

[braebo]

Productivity is a measure of progress towards a goal, not a judgment of whether that goal is worth pursuing.

[argee]

As Charlie Munger would say, something that isn’t worth doing at all sure as heck isn’t worth doing well.

[palmotea]

> So for somebody evaluating things based on what they do at work (where they're probably quite competent) or on personal projects well within their own domain, then it's 'hey what's all the hype about?' But if you're doing things outside your domain then it's a revolutionary game-changer.

They may also be seen as a revolutionary game-changer by people who actually kinda suck at their job.

And a lot of software engineers actually kinda suck at their jobs.

[jimmaswell]

> For things you're already highly skilled at LLMs can be handy but the overall gains, after all is accounted for, are not so clear.

Preposterous. I can tell 5.6 Sol to do something like "optimize this entire subsystem to have no ongoing memory allocations, fix these 5 bugs, implement these 2 features" and go work on other code or do something else entirely, and it does it all flawlessly. All things I could have done myself but nowhere near as fast and right the first time with such thorough test cases. We've reached the point where the AI is usually faster and always less effort overall compared to the old days, even if you have to send a few followup prompts.

[layer8]

> it does it all flawlessly

Serious question: How do you assess this? It seems to me that it would take very significant time to establish that conclusion.

[Lord-Jobo]

Validation is really hard and requires a great understanding of the entire relevant code. So to answer your question, they don’t know or asked the AI to do that too which has very obvious problems.

I find it hard to believe they would spend the effort and money making an AI do tasks like that, but are ALSO willing to spend the effort and time to properly validate the changes. I’m sure many do… but probably not a large percentage.

[jimmaswell]

In my case it's usually a video game in question, not a life support system, so I'm not poring over every line of code.

I mainly look at:

- The overall architecture to make sure it's in line with what I want

- The points where it interacts with certain other systems I'm concerned about

And I test the feature myself in addition to the test cases. Sol has proven to be outstanding at good test cases and having a coherent view on overall architecture integration. I basically never think to myself "this entire implementation is slop, I need to rip it out and refactor it" anymore.

I suspect the model's capacity to translate requirements into contextually appropriate test cases, and insistence on thoroughly covering functionality in tests, is a large part of its strength.

[bbg2401]

This is going to sound accusatory but I promise it isn’t.

Did you feel the same way about 5.5? The same general sentiment keeps being expressed with every model release. The prior generation is immediately cast aside with a vague “well yes, actually we didn’t mean it last time” attitude.

I think I even heard Theo the T3 guy suggesting that his work comes to halt if he can’t use the latest generation model, despite heaping significant praise on each generation previous and suggesting it does all of his programming tasks.

The boundless energy with which each model generation is said to be revolutionary and leaving everyone behind is tiresome even when incremental progress is real and measurable.

[jimmaswell]

Prior to 5.5 I thought the models were very useful but not trustworthy for certain tiers of tasks and needed more in depth verification. Frequent hand edits and times where it felt more productive to just finish something myself than keep rolling the dice on the model not subtly messing it up. But still great for targeted things.

5.5 on Extra High I found to be a huge leap in what it could tackle and how effectively but only up to a point. Still had common failure modes here and there but a big improvement

When I tried 5.6 I wasn't expecting more than an incremental upgrade, but it blew me away with how "smart" it seemed, how it completely sidestepped failure modes I'd been used to dealing with in 5.5, its success rate on the first try for wide-sweeping architectural tasks, and how I've basically never seen a bad hallucination make it through to the final output. It's the first time I've felt like if I left it with a task on the highest setting and it had any way to verify its output, and told it to keep going, it could solve almost anything. This could all be incremental upgrades in many different areas of the model+harness, but the cumulative effect feels like crossing that "escape velocity" threshold. The point where there are no more rough edges and it "just works".

I can't speak to others, but my feelings on all of these have been consistent since I tried them in the first place. I've also heard model providers draw people in at each new release with the model at full performance and then degrade it over time hoping users won't notice, which has seemed plausible anecdotally but I have no hard proof.

[orwin]

That's highly unlikely that this is done flawlessly. This morning I found a bug in our error handling, I asked Sol to fix it. He rewrote the way the specific component that failed called the exception. I asked it to retry and go deeper, inside the library. It solution was to ignore the part of the message that caused the issue. Which worked very well, but wasn't the root cause (it was a serialization issue. It's always a serialization issue). It couldn't go deeper, always stuck trying to catch the error rather than fixing it for real. We found the root cause, fixed it, and it was done, one line change and 15 lines of test, in one hours. Instead of four hours of nothing except spending tokens.

But truly I'm mad at the vscode harness. I used to be able to follow the 'thought' of the AI model and steer them when I saw a mistake, it's now way harder to do as multiple research tasks are done in parallel.

[jimmaswell]

Somehow I've never had an experience like that on Sol. I find it to never really "cheat" like that. I've had all my best results from Codex CLI though, maybe that's somehow different. I also don't go below xhigh for anything serious and if it's having an issue I escalate the thinking mode.

[pipes]

It has taken me a few months to get productive with them. I did not pick it up easily.