Yes. The other explanation is that time is nothing special, it gets just counted like everything else. You wouldn't say it's a quarter to a full cake either.
If we follow your argument, the number 3.25 would be read "point two five to four" istead of "three point two five". Which is to say, the fact that the quarters are mentioned in connection to the next hour, not the previous one, is indeed unusual.
Quarters are named by the hour they are in, neither the one previous or after. The hour number four of the day starts at 3:01 (or 3:00:01 with precision in seconds) and is complete by 4:00, it's the same mistake that people make with centuries.
'dreiviertel Vier' is short for 'dreiviertel der vierten Stunde des Tages'.