[ynniv]

The question is what does programming with an LLM get you over batteries-included frameworks with scaffolding like Rails or Django?

Three years ago an LLM would conversationally describe what the code would look like.

Two years ago it might crib common examples with minor typos.

Last year it could do something that isn't on StackOverflow at the level of an intern.

Earlier this year it could do something that isn't on StackOverflow at the level of a junior engineer.

Last week I had a conversation with Claude 3.5 that went something like this:

  Write an interactive command-line battleship game
  Write a mouse interactive TUI for it
  Add a cli flag to connect to `ollama` and ask it to make guesses
  There's a bug: write the AI conversation to a file so I can show you
  Try some other models: make options for calling OpenAI and Anthropic
  GPT and Anthropic are throwing this error (it needed to switch APIs)
  The models aren't doing as well as they can: engage them more conversationally

Elapsed time: a few hours. I didn't write any code. Keep in mind that unlike ChatGPT, Claude can't search the net for documentation - this was all "from memory".

What will LLMs do next year?

[mnky9800n]

I read these stories about using LLMs and I always wonder if it's survivor bias. Like I believe your experience. I've also had impressive results. But also a lot of times the ai gets lost and doesn't know what to do. So I'm willing to see it as a developer tool, but it's hard to see it become more general purpose in the next 6 months time frame people have been promising for the last two years.

[WhyOhWhyQ]

I played with it a year ago and it really hasn't improved much since then. I even had it produce a few things similar to your battle ship demo.

And next year I don't see it improving much either if the best idea anybody has it just to give it more data, which seems to be the mantra in ML circles. There's not an infinite supply of data to give it.

[ryandvm]

Absolutely. I posted a similar experience developing a Chrome extension with GPT 4o in a hour or so when it would have taken me at least a day to do on my own. I have no idea how people are hand waving LLMs away as no big deal.

I think the only justification for such a position is if you are a graybeard with full mastery of a stack and that's all you work in. I've dealt with these guys over the years and they are indeed wizards at Rails or Django or what have you. In those cases, I could see the argument that they are actually more efficient than an LLM when working on their specialty.

Which I guess is the difference. I'm a generalist and I'm often working in technologies that I have little experience in. To me LLMs are a invaluable for this. They're like pair programming with somebody that has memorized all of Stack Overflow.

[ruszki]

Where did you get that it can figure out things which was not feed into it (e.g. not on Stackoverflow)? In the past year, none could answer any of my questions, for which I couldn’t find anything on Google, in any reasonable ways. They failed very badly when there was no answer to my question, and the question should have been changed.