[mexicocitinluez]

> but they will not cover the user perspective or edge cases unless those are already documented somewhere

Are you suggesting that LLM's can't test for people who use screen readers? Keyboard only users? Slow network requests?

You're acting like the issues an app faces are so bespoke to the actual app itself (and have absolutely no relation to existing problems in this space) that an LLM couldn't possibly cover it. And it's just patently wrong.

[tovej]

I'm not talking about keyboards or screen readers or any sort of input testing, I'm talking about how the software is used in practice.

If you disagree with that, I think the onus is on you to show me that an LLM could simulate the full context in which a user interfaces with software. That's a ridiculous claim.

Feel free to show literally any evidence for this claim.

[mexicocitinluez]

I'm disagreeing with the saying it's impossible across the board, I'm not saying it's universally possible.

lol And you made the claim, not me. The proof is on YOU.

[tovej]

No, that's not how burden of proof works.

The status quo is that this capability does not exist. Whoever makes a claim contradicting the status quo has the burden of proof. I can't prove a negative.

And even with your logic, I did not make the original claim, it was made by simon.

Your statement now also makes little sense. For any nontrivial software project, the usage patterns and interactions with other systems are complex enough that the code itself does not contain enough context to understand how it is used, or what the invariants are.

There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare.

[mexicocitinluez]

> There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare.

Its not rare. I've built 2 dozen line-of-business apps in it last handful of years that were glorified CRUD apps. Every environment I've been in has had a mix of the 2.

And even then, that's at odds with your absolute above. On top of being in a field that changes daily.

[tovej]

You are interpreting a general statement as a categorical one.

I wasn't really going for an exact, formal statement, but I can give you a formal interpretation of what I said above, if you want to be pedantic.

In general, you can't expect an LLM to produce thorough documentation or robust tests for nontrivial software, because the use of those software (i.e. how their interfaces are expected to behave) contains assumptions from the context in which they are used, and that information will not be encoded in the source.

If the above was somehow ambiguous, this should be clear and uncontroversial.

[mexicocitinluez]

> You are interpreting a general statement as a categorical one.

That is in fact what I did and if you meant otherwise, then yes I agree that currently there are plenty of cases in which those tools fall short and will never replace a human.

[thunderfork]

>Are you suggesting that LLM's can't test for people who use screen readers? Keyboard only users? Slow network requests?

I don't think it's feasible to fully simulate the full depth of actual usage, given that (especially in the case of screen readers and the like) there's a great deal of combinatorial depth and context to the problem. Which screen readers, on which operating systems, and which users thereof?

[fibonacci_man]

I can’t tell if you’re being sarcastic or not

[mexicocitinluez]

You're saying that every app on this planet has bespoke usages that can't be derived from the app itself? That's your claim or am I getting this wrong?