I forget the exact wording in his autobiography but it was something like "I can't pretend I'm crazy enough to prefer being in my seat to the other two's. But I've got my own part to play."
They are also cut-throat competitive, ego maniacs. Just incredibly professional. I don't think you get to where those guys are without an incredibly appetite for success.
That's the central thesis of The Right Stuff. Collins was bumped from Apollo 8 and didn't feel that bad about bumping someone else from Apollo 11 as a result.