This is equivalent to asking what is the average number of digits necessary to represent an arbitrary natural number. If you assume that this is a finite number you can quickly arrive at a contradiction.
This limit diverges. Your intuition is breaking down. It might be useful for your understanding to study real analysis where these sorts of questions are handled rigorously.
> necessary to represent an arbitrary natural number
What makes you think this number exists? Infinities can not be added and divided this way.
> The average can't be larger than the largest number.
But it can be equal to it. You mistake is not realizing that infinity minus 1 = infinity.
So the average is "smaller" than the largest, and yet equal to it. Because like I said at the start, you can not just add infinities in the normal way you can manipulate finite numbers.