I think I read that it takes 80ms just to run the decryption algorithm on the iPhone, so while having unlimited retries would get them in eventually it will not speed up the brute force attack.
Assuming it is a 4 digit numeric passcode, that's only 10,000 possible combinations. At 80ms each (let's round up to 100ms to be conservative) that's only ~17 minutes. Even if it is a 6 digit pin we're only talking about ~28 hours (if they use a single, serial process).
That is true. Most people use a 4-digit or 6-digit numeric password and that's what the FBI is betting on when they ask to remove the brute force restrictions.
However Apple allows something like a 35-char max with numbers and letters so in the worst case they will never crack it.