| > Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. Proposed fix would probably be hacky, but so as Sonnet's I'll go on a tangent but to me that's what we're all seeing. It's the "record number of CVEs found by AIs" thing: these tools are extremely good at searching inside code. And that is a godsend. We' got people (claiming they're from Anthropic) posting comment saying: "Yes GLM 5.2 found that security bug in library xxx, but we just tried with Fable and it found it too". More code-searching, more bugs finding. Dick-measuring contests on bug finding abilities. But the headlines we don't see at all are: "1000 CVEs found by AI, 1000 CVEs fixed by code written by AI". These are nowhere to be found. We don't see "GLM 5.2 suggested an elegant fix to CVE-2027-xxxxxx" to then have a paid Anthropic shill posting "Fable suggested an eleganter fix than GLM 5.2". These headlines are, as of 2026, nowhere. You wrote the result would be "hacky". Here's why I saw from a top, paid for, SOTA model from the top company of the moment: instead of doing two integers comparison (literally one line of code) to verify that a value is between a range, the thing somehow noticed a "pattern" in the hexadecimal representation of the two values and went insane. It started converting the value to its hexadecimal string representation and then started doing substring string matching on that. "Hacky" is too nice of a word. This is pure garbage. Those who go hiking "while their agents ship features" don't realize the level of underperforming, buggy, insecure crap that their LLMs are generating. I found it very interesting the schism between those who use LLMs to find issues but who verify/modify or even don't use at all the fix they suggest and those who vibe-code while on a yoga retreat. It's 2026: LLMs do find bugs. But can they fix them? And do we even care: isn't finding a bug 99% of the job? |
> underperforming, buggy, insecure crap that their LLMs are generating
The biggest challenges with AI-generated code are: models actively destroy security features, Opus explained to me once that authorization mechanism is "bad development experience" all while making a backdoor (he made a skeleton key if token=="test" then all permissions granted). Also models actively destroy QA gates. I don't even complain when they delete tests - at least it's visible, they can flip condition to make a test pass, and with vast code changes these are hard to spot.
I myself, and some people I know "vibe-code" professionally though, but then we often assess not the code but it's behaviour. For instance, whether hand-made tests are all pass, p95 is under 50ms, and so on, I may not care about the implementation details.
On the other hand, my friend told me about garage owner he visited, 60 yrs old auto-mechanic, CRM, parts inventory management, payments processing terminal, passwords in txt, people's personal data God knows where, could be unprotected MySQL looking into the Internet bare for all we know.
2026 onwards will be wild.