[torginus]

My two cents is LLMs are way stronger in areas where the reward function is well known, such as exploiting - you break the security, you succeed.

It's much harder to establish whats a usable and well architected, novel piece of software, thus in that area, progress isn't nearly as fast, while here you can just gradient descent your way to world domination, provided you have enough GPUs.

[shepherdjerred]

Construction is always more expensive than destruction

[xpe]

To be pedantic: construction with an interconnected complex set of durable goals is hard. The general rule is that optimization over a constrained space is expensive.

But standing up a house of cards is pretty cheap. Examples include: shell corporations, formulaic business plans, AI slop, surface-level conversation, color by numbers, tract housing, cravenly only appealing the base desires of people, & c. (This might be the first time I've connected the dots in this way -- and it explains my distaste for all those things.)

But "cheap" isn't necessarily insecure. Installing bollards around building entrances is relatively cheap insurance against vehicular attacks. So this is more complicated than it seems. "Fast" doesn't mean unsafe. Even "hastily created" software _could_ be (relatively) secure if it was highly constrained to provably hardened patterns. A big problem comes when attacking a cheap target builds capability for the attacker. In a way, this analogous to how viruses attack. Start with an easy target, hijack the cell machinery, multiply, repeat.

Maybe this formulation is accurate?: If you creates something beyond your ability to understand it, then get ready to get pwned. "Staying in one's lane" in this sense might be 'safe' at least narrowly speaking (unless an entire industry is operating in a state of delusion, which is arguably the case now.)