[yjftsjthsd-h]

Obligatory: Either written by AI or by a human who has spent so much time with AI that they adopted its writing style. Anyways.

> Over 35 hours it performed 432 kernel evaluations. Each cycle meant writing code, compiling it, running it, reading the profiling output, deciding what to change, and trying again. The model diagnosed compilation failures it hadn’t seen before, identified performance bottlenecks through runtime feedback rather than prior knowledge, and redesigned the kernel architecture multiple times when incremental improvements stopped working.

Anyone remember genetic algorithms? This might be an improvement, but it still feels a little like deja vu.

[thatoneguy]

Yeah, I remember. I still have Usenet postings about the genetic algorims conference back in the '90s and some magazine clippings about researcher from the University of Sussex where I first learned about genetic algorithms back in high school.

[dist-epoch]

Genetic algorithm is random. This is intelligent evolution. Big difference.

[Kim_Bruning]

I got nerd-sniped wrt the genetic algorithm.

Technically birdshot from a shotgun is also randomly distributed (passing through a cone). This actually improves the chance of hitting the clay pigeon, because the birdshot spreads out and each individual ball has a chance to hit.

Genetic algo is similar. it's an optimizer that - in order to avoid local optima - will 'shotgun' an area around its current best guess.

[dist-epoch]

Yeah, but you shoot in the direction of the clay pigeon, you don't pick randomly a direction in space where to point your gun at.

[Kim_Bruning]

Compare genetic approach to greedy? With greedy approach you always take the next lower energy value no matter what, right? But if you end up in a local basin, you'll just never escape, because you never look further than your direct environs.

So instead of just sampling in a close 'circle' around your current point looking for a 'down', how about we spread that out a bit? You could use a 'circle' in a regular pattern, but what does that even look like in high dimensional space? Seems it's best to use some random distribution centered on your current position.

(LLMs actually have a 'temperature' setting which introduces noise for this exact reason.)

Some of GA's claims to fame are A) it uses purely just this distribution to descend. B) It can find multiple optima.

The way I think of it is that the simplest GA is basically greedy optimization with spread.

Greedy is like shooting a rifle , which is great for sniping, but you'll miss if the target is moving fast or doing things you can't quite keep up with.

A GA -like a shotgun- introduces spread: multiple chances to hit, multiple chances to escape local optima and rough patches in the landscape.

(A really good -if slightly morbid- modern example in the wild is COVID; which managed to outwit human civilization rather handily. "Not bad for a bit of encapsulated RNA" you'd think - until you realize it was running trillions of attempts in parallel. Really, the poor governments had no chance. )

[qrobit]

Both are non-deterministic, both have some metric to optimise, one is specific and efficient, the other is too broad and very expensive

[rurban]

Temperature is the rephrasing of randomness. No difference, just much better matchers.

[greenavocado]

Key word is non-differentiable optimization. That's what generic algorithms were traditionally good at.

[Zardoz84]

So a LLM wrote 432 kernel variations and it found what was the faster...