[aaroninsf]

For me this is a prototypical example of compounded cognitive error colliding with Dunning-Kruger.

We (all of us) are very bad at non-linear reasoning, reasoning with orders of magnitude, and (by extension) have no valid intuition about emergent behaviors/properties in complex systems.

In the case of scaled ML this is quite obvious in hindsight. There are many now-classic anecdotes about even those devising contemporary scale LLM being surprised and unsettled by what even their first versions were capable of.

As we work away at optimizations and architectural features and expediencies which render certain classes of complex problem solving tractable by our ML,

we would do well to intentionally filter for further emergent behavior.

Whatever specific claims or notions any member has that may be right or wrong, the LessWrong folks are at least taking this seriously...