[OtherShrezzing]

I think in the relatively near future we’re going to start seeing sophisticated supply chain attacks into language model training data.

It should be feasible to design vulnerabilities which look benign individually in training data, but when composed together in the agent plane & executed in a chain introduce an exploit.

There’s nothing technical really stopping that from existing right now. It’s just that nobody has put the effort in yet.

[lacunary]

The develop-test-refine feedback loop for this kind of attack is so long (or expensive) that it seems likely to limit its real world use. Poison training data, wait months? a year? for the model to come out, see how well it worked, refine... or do you see a faster way to iterate?

[OtherShrezzing]

Continual learning is the next major architectural milestone for the frontier labs. That’d reduce the iteration loop to days instead of years.

If your attacker assumes that all or most software will be generated from language models, the time penalty is worth paying.