In set phrases it can refer to things of various colors, but on its own it means blue. Like if you use it to describe a sweater, people will assume you mean a blue sweater.
If those words and others were more obscure I’d agree with your point, but they really are common enough that I think Aoi can mean blue or green in nodern Japanese.
Obscurity isn't involved. In set phrases like you're talking about, the "ao" is basically a category specifier, not a color specifier. Like aoba means young fresh leaves, not necessarily green ones, and aoyasai doesn't only mean green vegetables, etc. There are lots of set phrases like this, both obscure and common, and the color involved can be anything from white to dark gray.
But when ao is used outside of a set phrase, to tell somebody what color you're talking about, it means blue.