[GolDDranks]

Just a supplementary fact: I'm in the beneficial position, against the AI, that in a case where it's hard to provide that automatic feedback loop, I can run and test the code at my discretion, whereas the AI model can't.

Yet. Most of my criticism is not after running the code, but after _reading_ the code. It wrote code. I read it. And I am not happy with it. No even need to run it, it's shit at glance.

[elevation]

Yesterday I generated a for-home-use-only PHP app over the weekend with a popular cli LLM product. The app met all my requirements, but the generated code was mixed. It correctly used a prepared query to avoid SQL injection. But then, instead of an obvious:

    "SELECT * FROM table WHERE id=1;" 

it gave me:

    $result = $db->query("SELECT * FROM table;");
    for ($row in $result)
        if ($["id"] == 1)
            return $row;

With additional prompting I arrived at code I was comfortable deploying, but this kind of flaw cuts into the total time-savings.

[ReverseCold]

> I can run and test the code at my discretion, whereas the AI model can't.

It sounds like you know what the problem with your AI workflow is? Have you tried using an agent? (sorry somewhat snarky but… come on)

[GolDDranks]

Yeah, you're right, and the snark might be warranted. I should consider it the same as my stupid (but cute) robot vacuum cleaner that goes at random directions but gets the job done.

The thing that differentiates LLM's from my stupid but cute vacuum cleaner, is that the (at least OpenAI's) AI model is cocksure and wrong, which is infinitely more infuriating than being a bit clueless and wrong.

[storystarling]

I've been trying to solve this by wrapping the generation in a LangGraph loop. The hope was that an agent could catch the errors, but it seems to just compound the problem. You end up paying for ten API calls where the model confidently doubles down on the mistake, which gets expensive very quickly for no real gain.

[yaur]

Give Cluade Code a go. It still makes a lot stupid mistakes, but its a vastly different experience from pasting back and forth with chat gpt.

[tayo42]

There's no free trial or anything?

[yaur]

You can play with the model for free in chat... but if $20 for a coding agent isn't effectively free for use case it might not be the right tool for you.

ETA: I've probably gotten 10k worth of junior dev time out of it this month.

[tayo42]

The chat is limited and doesn't let you use the latest model. if that's representative of the answers I would get by paying, it doesn't seem worth it.

Im not crazy about signing up for a subscription service, it depends on you remembering to cancel and not have a headache when you do cancel.

[__MatrixMan__]

You might get better code out of it if you give the AI some more restrictive handcuffs. Spin up a tester instance and have it tell the developer instance to try again until it's happy with the quality.