[microtonal]

I think it all hinges on recognizing what opportunities automation helps.

For instance, I had to rename a collection of files almost following a pattern. I know that there are apps that do this and normally I’d reach for the Perl-based rename script. But I do it so irregularly that I have to install it every time, figure out how I can do a dry run first, etc. Meanwhile, with the Raycast AI integration that also supports Finder, I did it in the 10-15 seconds that it took to type the prompt.

There are a lot of tasks that you do not do often enough to commit them fully to memory, but every time you do them it takes a lot of time. LLM-based automation really speeds up these tasks. Similar for refactors that an IDE or language server cannot do, some kinds of scripts etc.

On the other hand LLMs constantly mess up some algorithms and data structures, so I simply do not let LLMs touch certain code.

It’s all about getting a feeling for the right opportunities. As with any tool.

[brendoelfrendo]

> For instance, I had to rename a collection of files almost following a pattern. I know that there are apps that do this and normally I’d reach for the Perl-based rename script. But I do it so irregularly that I have to install it every time, figure out how I can do a dry run first, etc. Meanwhile, with the Raycast AI integration that also supports Finder, I did it in the 10-15 seconds that it took to type the prompt.

> On the other hand LLMs constantly mess up some algorithms and data structures, so I simply do not let LLMs touch certain code.

See, these two things seem at odds to me. I suppose it is, to a degree, knowledge that you can learn over time: that an LLM is suitable for renaming files but not for certain other tasks. But for me, I'd be really cautious about letting an AI rename a collection of files, to the point that the same restrictions apply as would apply to a script: I'd need to create the prompt, verify the output via a dry run or test run, modify as necessary, and ultimately let the AI loose and hope for the best.

Meanwhile, I probably have a script kicking around somewhere that will rename a batch of files, and I can modify it pretty quickly to match a new pattern, test it out, and be confident that it will do exactly what I expect it to do.

Is one of these paths faster than the other? I'm not sure; it's probably a wash. The AI would definitely be faster if I was confident I could trust it. But I'm not sure how I can cross that threshold in my mind and be confident that I can trust it.

[saxenaabhi]

> See, these two things seem at odds to me. I suppose it is, to a degree, knowledge that you can learn over time: that an LLM is suitable for renaming files but not for certain other tasks. But for me, I'd be really cautious about letting an AI rename a collection of files, to the point that the same restrictions apply as would apply to a script: I'd need to create the prompt, verify the output via a dry run or test run, modify as necessary, and ultimately let the AI loose and hope for the best.

Why? I never understand this level of caution since don't we all use VC? Just feed it the prompt and if it messes up undo the changes.

[acuozzo]

> don't we all use VC?

This assumes you're working with text files.

What if you're working with ~100MiB (each!) frames from a scan of a 35mm movie?

(Note: This isn't fictional. I've worked with file-sets like this in film restoration many times.)

[brendoelfrendo]

As another commenter suggested, this only works for some workflows. I'd also argue it kind of undermines the idea that an LLM can do this work better than a script.

[gmadsen]

as part of the prompt, have a test suite with test files. Its still fully automated by the LLM but adds confidence

[recursive]

If it's under the umbrella of LLM automation, then I'd also need to verify that the test suite behavior actually matches the "production" behavior.

[gmadsen]

sure, but that is less work. you can also have separate LLM QA prompts that assess test suite behavior to production behavior.

ultimately you are right, the buck needs to stop somewhere, but at least in my experience, the more you add quality/test checks as LLM workflows, the higher the rate of success.