Having changed layout when I started learning Russian (since russian keyboard has letters in different places than their phonetic latin counterparts) I think it's just matter of how you learn. I had problems too, until I started learning by typing the same sentences that I used to learn Russian alphabet, which introduce letters in small batches of 3 or 4. After few weeks I was able to comfortably switch between both layouts every few sentences.
That's the main issue. I switched to Dvorak a year ago and it's been great, but being on other machines is difficult. The most egregious issue I've seen is a colleague who took the GRE and was forced relearn QWERTY. The best writer I've ever met score lower on that portion of the test due to keyboard layout.
That said, I'm happy I switched. Keyboard shortcuts in design applications took longer to mentally remap than the layout itself for daily typing. There are trade-offs (as with everything).