Its not the size but rather intelligent entity density. You could have an infinite universe but still have high ratio of contact if the universe has a high density of life forms (think of it as collisions with particles in a large volume).
Compared to the reducing human reproductive rate this is an important point. It's quite likely that somewhere in the regime of 12 billion is all the alive humans there ever will be - even with immortality technology. After all, how many times per 1000 years would you want to have some more children? How many of those people are likely to exist compared to a dormant mass who don't have children?
Now factor in a lack of FTL travel. If it takes thousands of years to go exploring, how many explorers are there likely to be, and how likely are they to stop and setup colonies instead?
Yes this is similar to the self replicating probe problem (one solution to the SETI problem is to launch probes that replicate). That probes start having the same problems as the creating intelligent entity has. Either the probes can't replicate fast enough or decide to stop replicating/exploring for whatever reasons and there are just not enough/fast enough probes.
> For me the best explanation is the size of the Universe
Size, and time of the Universe. Most probably species do not last beyond a couple of million years no matter what, and there may be very little overlap between periods of intelligence even if they were in relatively close area of space.