[dagenleg]

If the population does not want to be part of the country, one cannot just annex the territory in the modern times.

[ceejayoz]

Correct. That said, if the UK leaves the EU, and freedom of movement goes away, the opinions of Gibraltar's residents may change fairly rapidly.