The difference between 32 bits and 64 bits is the amount of people on Earth compared to (EDIT) the amount of grains of sand on Earth. 32 bits is nothing when it comes to entropy, and it can take a security researcher (like us) only $100 to rent a machine to completely brute force it. Nowadays, only values less than 128 or 256 bits (which are exponentially bigger) are seen as appropriate.
That is correct, time is the only entropy for the command, and the function they use to generate random numbers is also flawed in that it can only produce 2^32 possible outputs.
Also if it's really the date it's nowhere near 32 bits of entropy. I'm guessing you can pretty easily guess to the day when a Bitcoin wallet was created, so that's about 16 bits of entropy. Less if you know the time, possibly 0.
It actually uses the most precise 32 bits of the date, so it's any, like, nanosecond between 0 and some other small amount of seconds. You can't brute force a wallet by knowing approximately when it was made, but you can brute force every mnemonic if you have the time or a bit of cash to throw at a server.
EDIT: It loops around to 0 every 4.something seconds, so it's not like everything after 4 is the same key. It's just a more random distribution than what you may be thinking.
There is often very low entropy in the lowest few bits of system time as well (due to the underlying clock having a different resolution than the system call). Given that every bit you lose halves the time for a brute-force, that's a problem.