[miki123211]

If there's one thing LLMs are really, really good at, it's having a target and then hitting / improving upon that target.

If you have a comprehensive test suite or a realistic benchmark, saying "make tests pass" or "make benchmark go up" works wonders.

LLMs are really good at knowing patterns, we still need programmers to know which pattern to apply when. We'll soon reach a point where you'll be able to say "X is slow, do autoresearch on X" and X will just magically get faster.

The reason we can't yet isn't because LLMs are stupid, it's because autoresearch is a relatively new (last month or so) concept and hasn't yet entered into LLM pretraining corpora. LLMs can already do this, you just need to be a little bit more explicit in explaining exactly what you need them to do.

[eru]

> The reason we can't yet isn't because LLMs are stupid, it's because autoresearch is a relatively new (last month or so) concept [...]

I'm not so sure. People have been doing stuff like (hyper) parameter search for ages. And profiling and trying out lots of things systematically has been the go-to approach for performance optimisation since forever; making an LLM instead of a human do that is the obvious thing to try?

The concept of 'autoresearch' might bring with it some interesting and useful new wrinkles, but on a fundamental level it's not rocket science.

[philipallstar]

I've not tried this yet, but doesn't it use up loads of tokens? How do you do it efficiently?

[eru]

It uses a lot of minutes on your computer(s), since you need to run lots and lots of experiments.

I'm not sure if it's particularly token hungry.