[tivert]

> I believe it is almost certain that we will make something like this and that they will out-compete us. The bigger problem here is that too few people believe this to be a possibility. And when this becomes certainty becomes apparent to a larger set of people, it might be too late to tone this down.

I think the bigger problem is that too many people are focused on short term things like personal wealth or glory.

The guy who make the breakthrough that enables the AGI that destroys humanity will probably win the Nobel Prize. That potential Nobel probably looms larger in his mind than any doubts that his achievement is actually a bad thing.

They guy who employs that guy or productionizes his idea will become a mega-billionaire. That potential wealth and power probably looms larger in his mind than any doubts, too.

[hollerith]

That is why the government should help the researcher and the tycoon do the right thing by shutting down the AI labs and banning research, teaching and publishing about frontier AI capabilities.