[fighterpilot]

I think one thing to consider is the non-stationary distributions in the problem space.

We may be able to formulate our current understanding of what it takes to be a CEO in every feasible future context that we can imagine, and create an agent (not an AGI) that does that.

But then what would happen when the distributions change out of sample in an unforseeable way? Suppose two countries go to war and this drastically changes the operating environment of the business. The agent would need to learn how to operate with human-level capability in this novel environment which it wasn't trained specifically to do. That's why I'm thinking it requires an AGI.