Yes, this is what a CLA solves. And I tend to refrain from contributing to projects that have a CLA, although it might not be reasonable: there's little difference with licensing your contribution under MIT, I think.
However, if you license your project under AGPL, you probably deemed AGPL comfortable for your needs. I did too, so licensing my contribution under something else feels a bit uncomfortable, especially if it is substantial.
Nemo dat quod non habet, nobody gives what they don’t have.