Settings, Sounds, Headphone safety, reduce loud sounds, adjust the slider accordingly. I had to turn this on because my IEMs were way too loud on the lowest setting.
I had that on the lowest setting already, it doesn't make a difference. I suspect that iOS already thinks its playing music quite softly and therefore won't know to down correct.