[PaulDavisThe1st]

That's not how software development works.

Folks think, they write code, they do their own localized evaluation and testing, then they commit and then the rest of the (down|up)stream process begins.

LLM's skip over the "actually verify that the code I just wrote does what I intended it to" step. Granted, most humans don't do this step as thoroughly and carefully as would be desirable (sometimes through laziness, sometimes because of a belief in (down|up)stream testing processes). But LLM's don't do it at all.

[sally_glance]

They absolutely can do that if you give them the tools. Seeing Claude (I use it with opencode agents) run curl and playwright to verify and then fix it's implementation was a real 'wow' moment for me.

[Q6T46nT668w6i3m]

We have different experiences. Often I’ll see Claude, et. al. find creative ways to fulfill the task without satisfying my intent, e.g., changing the implementation plan I specifically asked for, changing tolerances or even tests, and frequently disabling tests.

[sally_glance]

Yeah I feel that, if it happens your only way out is to write down a more extensive implementation plan first. For me that is the point where I start regretting to have tried implementing something using AI,.. But admittedly most of the time redacting the implementation plan and running the agent again is still faster than I could have done on my own (I try to make implementation tasks explicit in the form of a markdown file, worked pretty well so far).

[Fr0styMatt88]

I see these “you had a different experience than me” comments around AI coding agents a lot and can concur; I’ll have a different experience with Copilot from day-to-day even, sometimes it’s great and other days I give up on using it at all it’s being so bad.

Makes me honestly wonder — will AGI just give us agents that get into bad moods and not want to work for the day because they’re tired or just don’t feel like it!

[ssl-3]

If part of the goal is to emulate a person's abilities, then surely that includes a person's ability to fuck things up.

[DANmode]

Are you a customer?

[DANmode]

Don’t downvote because you don’t like the question.

It obviously adds to the discussion: paid and non paid accounts are being conflated daily in threads like these!

They’re not the same tier account!

Free users, especially ones deemed less interesting to learn from for the future, are given table-scraps when they feel it’s necessary for load reasons.

[nineteen999]

Exactly. There's an impedance mismatch between those using the free/cheap tiers and those paying a premium, so the discussion gets squirrely because one side is talking about apples and the other oranges.

[DANmode]

Right.

More specifically: One side is talking about apples,

and the other is talking about mushy old apples,

that sometimes you need to wait 12 hours for.

[baobun]

All user accounts are also customers. Some are paying with data and contributing to metrics going up.

[DANmode]

That’s not how words work.

All users are stakeholders.

They’re emphatically not considered customers.

We can disagree with that, create legal protections for those people - but that doesn’t make them customers to OpenAI, Anthropic, et al.

[mapontosevenths]

> LLM's skip over the "actually verify that the code I just wrote does what I intended it to" step.

I'm not sure where this idea comes from. Just instruct it to write and run unit tests and document as it goes. All of the ones I've used will happily do so.

You still have to verify that the unit tests are valid, but that's still far less work than skipping them or writing the code/tests yourself.

[butlike]

I disagree it's less work. It just carte blanche rewrites tests. I've seen it rewrite and rewrite tests to the point of undermining the original test intention. So now instead of intentionally writing code and a new unit test, I need to intentionally go and review EVERY unit test it touched. Every. Time.

It also doesn't necessarily rewrite documentation as implementation changes. I've seen documentation code rot happen within the same coding session.

[mapontosevenths]

I've seen it do that as well. Especially Gemini 3 lately.

I've started to add an instruction to my GEMINI.md after I'm happy with the tests telling it not to edit them, but to still run them.

I solve the documentation issue the same way. By telling it when and what to update in the .md file.

[jimmaswell]

> actually verify that the code I just wrote does what I intended it to

That's what the author did when they ran it.

[adventured]

Claude Opus 4.5 will routinely test its own code before handing it off to you, even with zero instruction to do so.

[PaulDavisThe1st]

One commercial equivalent to the project I work on, called ProTools (a DAW), has a test "harness" that took 6 people more than a year to write and takes more than a week to execute.

Last month, I made a minor change to our own code and verified that it worked (it did!). Earlier this week, I was notified of an entirely different workflow that had been broken by the change I had made. The only sort of automated testing that would have detected this would have been similar in scope and scale to the ProTools test harness, and neither an individual human nor an LLM is going to run that.

Moreover, that workflow was entirely graphically based, so unless Claude Opus 4.5 or whatever today's flavor of vibe coding LLM agent is has access to a testing system that allows it to inject mouse events into a running instance of our application (hint: it does not), there's no way it could run an effective test for this sort of code change.

I have no doubt that Claude et al. can verify that their carefully defined module does the very limited task it is supposed to do, for cases where "carefully defined" and "very limited" are appropriate. If that's the only sort of coding you do, I am sorry for your loss.

[utopiah]

> access to a testing system that allows it to inject mouse events into a running instance of our application

FWIW that's precisely what https://pptr.dev is all about. To your broader point though designing a good harness itself remains very challenging and requires to actually understand what value for user, software architecture (to e.g. bypass user interaction and test the API first), etc.

[PaulDavisThe1st]

> Puppeteer is a JavaScript library which provides a high-level API to control Chrome or Firefox

my world is native desktop applications, not in-browser stuff.

[nineteen999]

You suggest a web testing framework as a response to someone working on a real desktop app?

[utopiah]

No I was sharing an example of a framework that does include "a testing system that allows it to inject mouse events".

That being said mouse events and similar isn't hard to do, e.g. start with a fixed resolution (using xrandr) then xdotool or similar. Ideally if the application has accessibility feature it won't be as finicky.

My point though was just to show that testing with GUI is not infeasible.

Apparently there is even a "UI Testing for devs & agents" https://www.chromatic.com which I found via Visual TDD https://www.chromatic.com/blog/visual-test-driven-developmen... I can't recommend this but it does show even though the person I was replying with can't use Puppeteer in their context the tooling does exist and the principles would still apply.

[PaulDavisThe1st]

> My point though was just to show that testing with GUI is not infeasible.

Indeed, which is why I mentioned the ProTools test harness and the fact that it took 6 people a year to write and takes a week to run (or took a week, at some point in the past; it might be more or less now).

[astrange]

Claude can do that, yes.

https://platform.claude.com/docs/en/agents-and-tools/tool-us...

Although if you want to test a UI app, it's better to do it through accessibility APIs rather than actually looking at the screen and clicking.