In the US, yes. And if it's a mixture of olive and something else they also just can't call it "olive oil" on the front label either.
On the other hand that's just what's required. If nobody's looking closely, who's to say what you're actually doing, or what has happened from your upstream suppliers with or without your knowledge?
> Do they not have to declare the other ingredients on the packaging?
They do but it's the same with everything else: there are definitions of what is something and sometimes they allow unexpected ingredients or mixtures that we wouldn't have expected.
I think milk or milk-based is one of those things. Same for beef burger (if it's 62% beef than it's allowed to be caled a beef burger, no matter if it tastes like cardboard).
On the other hand that's just what's required. If nobody's looking closely, who's to say what you're actually doing, or what has happened from your upstream suppliers with or without your knowledge?