It's also your tax ID, unless you've filed for a separate one. While the original intent may have been that the SSN would only be used for Social Security, by now various places are required by law to collect your SSN even if they have nothing to do with the Social Security Administration. But that isn't even the real problem here. SSNs are an identifier (username). Merely knowing someone's SSN shouldn't grant you any extra privileges. It's not a secret—lots of people know it, and it isn't even hard to guess from age and birthplace—and yet people treat it as if it were something akin to a password, as if knowledge of a person's SSN were enough to establish identity. By this point in time we should very well know to use only zero-knowledge proofs for authentication and not even share the secret with the entity one is authenticating to, so they can't turn around and impersonate you to someone else.
The same goes for credit card and bank account numbers. As anyone you've ever paid with your card or by check has access to these they can't be considered particularly secret. The problem is that the system barely has any authentication built in. 4-digit PINs and hastily handwritten signatures only an expert can verify do not offer reasonable security. Chip cards are a bit better (non-clonable, require physical possession) but only work for in-person transactions.