The argument is friends' contact info belongs to your friends, not to you. Its a social grey area that has been created by the internet, the correct protocols have yet to be defined.
I can access my friends contact info (including email addresses and phone numbers) as I like, within FB's platform. FB just restricts me from accessing that once I;m not longer connected to it. Indeed, you're right - that's the argument - but I suspect it's merely used as a justification to FB's commercial benefit.
Facebook has several features requesting users to add not only their username but passwords to other websites, to add friends. The same can not be said of Facebook [0]. The CEO is allegedly said to take advantage of journalists who provided their details to access journalists' private emails [1].