[usrusr]

That scenario in the last paragraph is an uncomfortably convincing view of the future. Do you think it could be avoided by an additional step like the following?

Step 5: a mechanism for curated metamodules

This would not have changed much for the current situation, but it might help cultivating a "too important to play it fast and loose" mindset that would.

[spenuke]

No, probably not. On the contrary, what I'm suggesting is that the solution of Step 5 you suggest would be "the hot new thing" of Step 4.

It's about unforeseen consequences of a solution to a problem. That's what I mean by "I don't know what it will look like", because it will definitely solve the problem this thread is dissecting, and the problem it introduces will be a whole new vector.

But its selling point (the way it solves our current problem) will be a simple idea that is taken to its logical extreme by people who don't think critically, and then it will be time to solve that problem.

That is, I see the underlying recurring problem as stemming from the cultural forces –– how we educate people, how we make ourselves hireable –– that enable even very smart people to be shortsighted and immune to learning from the past.