I wondered this too. I'm sure you could narrow the universe to some degree. Most wallets use the BIP 39 word list as a seed to derive the private key. So 12 words (in order) out of 2048 possible words is probably less than the total number of possible bitcoin addresses. This is just one example.