[floppydisk]

parent has some great points. You only will really learn by doing.

A couple of things you can do is start reading up on well designed systems and the gotcha stories. Personally, I track highscalability.com since I spend a lot of time working in distributed systems. They tend to have interesting insights and will link to tech talks and blogs that contain a wealth of information. Also go read up on some of the papers explaining the designs behind systems you use (e.g. Amazon dynamo) and why they made the choices they did. It'll give you some insight into how people measure tradeoffs and figuring out what's important.