[niceairport]

I have mixed feelings between your response and the response you are replying to. One one hand I can clearly see how what you're saying makes sense. On the other, I still am inclined to believe that if you really distilled the actual purpose of everything you mentioned, it could be formulated in a way that doesn't require AGI.

[fighterpilot]

I think one thing to consider is the non-stationary distributions in the problem space.

We may be able to formulate our current understanding of what it takes to be a CEO in every feasible future context that we can imagine, and create an agent (not an AGI) that does that.

But then what would happen when the distributions change out of sample in an unforseeable way? Suppose two countries go to war and this drastically changes the operating environment of the business. The agent would need to learn how to operate with human-level capability in this novel environment which it wasn't trained specifically to do. That's why I'm thinking it requires an AGI.