Unfortunately Facebook does its best to track those of us who don't use it as well (even if their information on us ends up being less comprehensive than the information they have on their actual users).
And you can be pretty effective at that tracking if your contacts use facebook. Since they have access to your call logs they know how frequently you communicate with someone. And people that communicate frequently share general interests.