Yea, it is such weird behavior. I can count the number of times I've ever wanted to adjust the volume of the ringer, rather than just turn it on/off, on one hand.
Luckily on newer version of Android there is always an option to quickly access the media volume after you press the volume button once, but it's still unintuitive.
Windows phone nailed this. As you change volume, a panel comes in with two sliders: one media, one ringer, so if you were in the wrong context and changed the wrong volume, you right away have access to the other one. I find (on iOS now) I fiddle with the ringer volume a lot throughout the day. Loudest possible is waaaay too loud for me, unless I’m in a noisy place.
Luckily on newer version of Android there is always an option to quickly access the media volume after you press the volume button once, but it's still unintuitive.