I'm Swedish too, and grew up with the same understanding as the parent poster. Only in recent years have I understood that khaki can refer to both a military type green or sand like beige color.
Swede here. I asked my wife too, her reply is that the color "khaki" is a shade of green - "khakigrönt". But the specific style of pants - "khakibyxor" - are generally beige.