I'm doing something similar to the OP but I'm not a CTO but a "manager". Maybe the title is "we're all managers now". The CTO role is bigger than that.
And, we are not managing the LLMs. We are communicating with them. The better a developer gets it communication in general, the easier their life will be in the better job they will do for their company. The more they understand about what their customers need, the better job they will do building those tools. None of this is about managing. It's about learning how to communicate, how to ask the right questions and how to listen.