This is actually what I do. I'm extremely picky about the code and force the LLM to rewrite it 1000x times until it is basically exactly what I want. You might be wondering what is the point when it would be faster for me to just write the code myself?
I have ADHD and for whatever reason telling the LLM what to do instead of doing it myself bypasses the task avoidance patterns and/or focus problems I tend to suffer from. I do not find it fun, but I am thankful for it.
I have used LLMs a couple of times to get started on something. I don’t have ADHD, so this is not a regular occurrence for me. But when I have tried this, I have always found the LLM solution so horrible that it instantly inspired me to do it myself. So, in that sense it worked, I got unstuck, but no LLM garbage makes it into the project.
That’s how I use it for writing. I am looking for alternate wording/phrases that I usually don’t use (language habits), alternate takes of any quality just to get myself thinking along different lines, etc.
Rarely do I use what the tool actually spits out. I just use it as a sounding board, like I’m chatting with a (very noob) writer. It doesn’t make me much faster but it helps me break through when I just can’t get words down.
But what if the only real way to break through avoidance patterns is to stop avoiding? What if the tradeoffs of LLMs are instant gratification and further atrophying of your executive functions?
I have gained a paranoid suspicion that our capacity to decrease immediate distress with technology has become so great that we are creating a world where people with certain temperaments can have their personalities become more and more extreme through the assistance of technologies which, for example, decrease the amount of interpersonal interaction required or prevent the need for deep focus.
This framing of it being a tool that you find indispensable as an individual is important. I’m not interested in debating static vs dynamic types, or vim vs emacs, etc. If it works for you, then that’s great!
But the difference with LLMs currently - I guess? - is that non-engineers are pushing the idea that it’s universally indispensable at scale. I think it leads to a lot of emotion bleeding into the debate.
This is what I'm seeing - for people who were slow and didn't posses a lot of depth or breadth, their blast radius and impact has skyrocketed. They can now work in unfamiliar domains quickly, without any knowledge of the nitty gritty details of those domains!
For me personally, it's a tradeoff of generating the first pass code 10x more quickly, but then deeply knowing and validating the code is then 10-20x more work than it would have been if I'd written it myself (and if time is of the essence, then there's the option of shallow validation/understanding in exchange for speed - which is a compromise in rigor and path towards tech debt). In the end, none of this seems like a net win (unless you don't care about quality), and it is much less enjoyable.
TL;DR; While LLMs are faster to spit out first pass code, by the time I've validated and fixed the LLM's first-pass work, I could've had my "by-hand" implementation done correctly, and had much deeper understanding out of the box. Net loss.
Even if it does speed me up, coding with LLMs suck all the joy out of writing software. Constantly babysitting the agent(s) and reviewing their output carefully is probably more exhausting than just writing things myself..
It is significantly easier to micro-manage an AI than a suite of junior developers. The AI doesn't replace a principal engineer, it's replacing junior and weaker senior developers who need stories broken down extremely concisely to be able to get anything done. The time it takes to break down a story such that a junior through weak senior developers can pick it up and execute it well would have the AI already done with testing built around it.
Juniors learn. Some juniors are potential good seniors. Over time they will internalise good architecture and be able to make good judgments on their own.
Micromanaging LLMs is like having Dory from Finding Nemo as your colleague. You find ways to communicate, but there is no learning going on.
LLMs can learn, just not the same way that juniors do. When an LLM does something wrong you can always update it's rules or skills to not make that mistake again. Or you can utilize a subagent whose sole purpose is to review code to prevent that mistake. Lots of ways you can improve LLMs over time.
Of course if you don't provide that feedback loop, no learning happens. I guess the same could be said of a junior, though.
Building larger systems of accountability isn't usually what people mean by learning. And besides, if telling an LLM not to do something were actually reliable, then LLMs would be a lot more useful than they are. And even if that were reliable, then you're just reinventing expert systems, which didn't work.
I'm not sure the point of contention is whether or not an arbitrary language model is capable of understanding new concepts and not make the same mistake again, as it is being used.
When people compare LLMs to juniors it's "can I have it do something pretty brain numbing, and when it makes mistakes can I invest time into preventing that from happening again, either systemically or via training?"
IME this is true for LLMs, at least in how my team has been utilizing them. This doesn't make juniors worthless, as they can be useful for things that LLMs aren't good at.
No, but you can fire them. Can you fire an AI you're paying for? Yes, but your options are another AI that is just as bad, or worse.
And it's really someone's fault for hiring a bad junior. Someone did interview them, right? Maybe the person that hired them is the problem. And maybe the person that decided to go all-in on AI is also the same problem.
I think if you tried working with some junior folks, you'd be quite surprised. You know, with at least some of them choosing to use their brains and all.
Honestly, I think so. I do a mix of infrastructure and programming so don't tend to have any frameworks memorized. Using AI is much quicker than constantly referencing the docs.
I can also switch between codebase with different frameworks and languages and make changes without spending all day reading docs.
It's also pretty good at tracing code and that's fairly straight forward to verify the results manually. It can build a flow diagram in 10-30 minutes (depending on what tool calls need allowed and how many prompts it needs) versus me taking a couple hours to do the same.
I don't micromanage it. I let my projects custom linter micromanage it.
Every project should have a custom linter for their tech stack. It would check for not just syntax errors, but architectural choices as well as taste guidelines.
Whenever the LLM writes bad code, I add it to my linter to check against in the future.
I have ADHD and for whatever reason telling the LLM what to do instead of doing it myself bypasses the task avoidance patterns and/or focus problems I tend to suffer from. I do not find it fun, but I am thankful for it.