[ramparrt]

I think it's because most hiragana/katakana is in the form "consonant-vowel" meaning that almost all words are made up of collections of 2 letter sounds/syllables. This gives words a certain cadence which is easily repeated. When you convert that to romaji you begin to parse the words as though they were english and the discrete "consonant-vowel" sounds get broken down and become harder to see/read.

Really though, you can learn hiragana in a single sitting using mnemonic trickery and you can commit it to memory using a spaced repetition or flashcard system in a few days. Not learning hiragana and katakana seems much more of a hindrance than "wasting time" by learning it.