It also has to make the networking part of Bitcoin fit within physics. From eyeballing the difficulty graph it looks like if you fixed the network difficulty in January 2010 then a new block would be mined every few picoseconds at the peak hash rate this year.
In the case of a fork in the chain the network decides the winner based on which fork ends up being longer. If blocks are being mined too quickly then forks become more likely and the state of the chain is more volatile as the current "winner" keeps flipping around.
Blocks being mined too slowly is a more obvious problem: less transaction throughput.