This is a limitation that I haven't really found a way through. I haven't tried too hard though because I've found that I can often just let the complicating set up have the 'plain' github.com host and use the custom host for the other set up. If I found that both professionally and personally I needed submodules, though, I'd probably be in trouble.
Also, if the submodules are public, the plain host will work fine because it doesn't matter what key you authenticate with.
Also, if the submodules are public, the plain host will work fine because it doesn't matter what key you authenticate with.