[basdw]

As Tomtoise said, they indeed read your address book. Even if you do not use the messenger app Whatsapp. You have given permission to let them read your whole phone (sms, address book and even to make changes in your calendar without your consent), by installing it. That is the main reason why many people do not install the Facebook app, but only use it via a browser.

[talltofu]

I actually just checked my address book and it seems I have never saved his phone number as a contact! The only other explanation could be that he saved my phone number as a contact and let Facebook have a field day with it