Yeah but another idea is that it is isotropic, which is borne out by observation. This leaves a few obvious possiblities:
1. It's infinite and homogeneous
2. It's finite but periodic (like the Earth; go straight long enough and end up where you were)
3. It's slowly varying, so that within our Hubble volume (the volume within the horizon receding from us at the speed of light, beyond which we can't see) everything looks isotropic; in other words, it varies, but slowly
So what you're saying is true from our position in space, but it should also (if points 1 and 2 are true) be true everywhere in space, and the question of the large scale structure of the universe remains valid.