[nozzlegear]

Last night I asked ChatGPT 4 to help me write a quick bash script to find and replace a set of 20 strings across some liquid files with a set of 20 other strings. The strings were hardcoded, it knew exactly what they were in no unclear terms. I just wanted it to whip up a script that would use ripgrep and sed to find and replace.

First, it gave me a bash script that looked pretty much exactly like what I wanted at first glance. I looked if over, verified it even used sed correctly for macOS like I told it, and then tried to run it. No dice:

    replace.sh: line 5: designer_option_calendar.start_month.label: syntax error: invalid arithmetic operator (error token is ".start_month.label")

Not wanting to fix the 20 lines myself, I fed the error back to ChatGPT. It spun me some bullshit about the problem being the “declaration of [my] associative array, likely because bash tries to parse elements within the array that aren’t properly quoted or when it misinterprets special characters.”

It then spat out a “fixed” version of the script that was exactly the same, it just changed the name of the variable. Of course, that didn’t work so I switched tactics and asked it to write a python script to do what I wanted. The python script was more successful, but the first time it left off half of the strings I wanted it to replace, so I had to ask it to do it again and this time “please make sure you include all of the strings that we originally discussed.”

Another short AI example, this time featuring Mistral’s open source model on Ollama. I’d been interested in a script that uses AI to interpret natural language and turn it into timespans. Asking Mistral “if it’s currently 20:35, how much time remains until 08:00 tomorrow morning” had the model return its typical slew of nonsense and the answer of “13.xx hours”. This is obviously incorrect, though funnily enough when I plugged its answer into ChatGPT and asked it how it thought Mistral may have come to that answer, it understood that Mistral did not understand midnight on a 24 hour clock.

These are just some of my recent issues with AI in the past week. I don’t trust it for programming tasks especially — it gets F# (my main language) consistently wrong.

Don’t mistake me though, I do find it genuinely useful for plenty of tasks, but I don’t think the parent commenter is wrong calling it snake oil either. Big tech sells it as a miracle cure to everything, the magic robot that can solve all problems if you can just tell it what the problem is. In my experience, it has big pitfalls.

[codexon]

I have the same experience. Every time I try to have it code something that isn't completely trivial or all over the internet like quicksort, it always has bugs and invents calls to functions that don't exist. And yes, I'm using GPT-4, the best model available.

And I'm not even asking about an exotic language like F#, I'm asking it questions about C++ or Python.

People are out there claiming that GPT is doing all their coding for them. I just don't see how, unless they simply did not know how to program at all.

I feel like I'm either crazy, or all these people are lying.

[nkozyra]

> I feel like I'm either crazy, or all these people are lying.

With some careful prompting I've been able to get some decent code that is 95% usable out of the box. If that saves me time and changes my role there into code review versus dev + code review, that's a win.

If you just ask GPT4 to write a program and don't give it fairly specific guardrails I agree it spits out nearly junk.

[oblio]

> If you just ask GPT4 to write a program and don't give it fairly specific guardrails I agree it spits out nearly junk.

The thing is, if you do start drilling down and fixing all the issues, etc, is it a long term net time saver? I can't imagine we have research clarifying this question.

[taco-hands]

> People are out there claiming that GPT is doing all their coding for them. I just don't see how, unless they simply did not know how to program at all.

I doubt it and certainly not for anything beyond basic. I've seen (and tried GPT's for code input a lot) and often they come back with errors or weird implementations.

I made one request yesterday for a linear regression function (yes, because I was being lazy). So was chatGPT... It spat out a trashy broken function that wasn't even remotely close to working - more along the lines of pseudo code.

I complained saying "WTH, that doesn't even work" and it said "my apologies" and spits out a perfectly working accurate function! Go figure.

Others have turned to testing tips or threats, which is an interesting avenue: https://minimaxir.com/2024/02/chatgpt-tips-analysis/

[throwawayanothe]

Same experience here

[jstummbillig]

I hear you. It's all pretty bad. I have spent half-days getting accustomed to and dealing with gpt garbage — but then I have done that plenty of times in my life, with my own garbage and that of co-workers.

On the margins it's getting stuff good enough, often enough, quick enough. But it very much transformed my coding experience from slow deliberation to a rocket ride: Things will explode and often. Not loving that part, but there's a reason we still have rockets.

[oblio]

Most of us don't ride nor want to ride rockets. We do use them to blow each other up, statistically.

[jstummbillig]

> Most of us don't ride nor want to ride rockets.

The amount noise generated by pretty much anything new and shiny on this website would disagree with that.

> We do use them to blow each other up, statistically.

Very true — and yet :^)

[foobiekr]

I've had the same experience, but I usually get what I want. Admittedly, I'm using it to script around ffmpeg which is a huge pain in the ass.

That said, every single script it churns out is unsafe for files with spaces on the first go round. Like.. Ok. It's like having a junior programmer with no common sense available.