[ceejayoz]

Short version: look at a map.

Long version: https://en.wikipedia.org/wiki/Disputed_status_of_Gibraltar

[dagenleg]

If the population does not want to be part of the country, one cannot just annex the territory in the modern times.

[ceejayoz]

Correct. That said, if the UK leaves the EU, and freedom of movement goes away, the opinions of Gibraltar's residents may change fairly rapidly.