The argument is friends' contact info belongs to your friends, not to you. Its a social grey area that has been created by the internet, the correct protocols have yet to be defined.
I can access my friends contact info (including email addresses and phone numbers) as I like, within FB's platform. FB just restricts me from accessing that once I;m not longer connected to it. Indeed, you're right - that's the argument - but I suspect it's merely used as a justification to FB's commercial benefit.
Facebook has several features requesting users to add not only their username but passwords to other websites, to add friends. The same can not be said of Facebook [0]. The CEO is allegedly said to take advantage of journalists who provided their details to access journalists' private emails [1].
As is the way it should be; by befriending you on Facebook I'm not saying that you are free to import my contact information into any old service that may use it to spam me.