[_pdp_]

Take some working code. Ask an LLM to fix bugs. Measure performance and test coverage. Feed the results back into the LLM. Repeat.

This has been the standard approach for more complex LLM deployments for a while now in our shop.

Using different models across iterations is also something I've found useful in my own experiments. It's like getting a fresh pair of eyes.

[cyanydeez]

Can we modify this approach to get LLMs that are good at specific programming languages or frameworks? That seems to be where local LLMs could really shine.

[nico]

Would love to have a small local model that only knows about rails and mvc web development

Alternatively, a modular model with multiple “experts” that I could mix and match for my specific stack

I don’t need the model to know all of the Internet plus 20 different human languages. I just want it to be really good with the stack of the project

[mememememememo]

LLMs shine through emergent behaviour. Finding an LLM that does Rails doesn't know poetry is like finding a Rails human developer who doesn't have a hobby e.g. basketball. So what if they play basketball? They can code too!

[nico]

Then it might need a new type of architecture to work. I’m not attached to LLMs. If a new model comes out that can do only the things I want it do it, then great

[mememememememo]

Sure. I'd use it too. I am just not sure they have that trade off to make. Yet (maybe with more research)

[barrenko]

It's just RL-everything.