It almost certainly already is, at least in some jurisdictions for some forms of data. GDPR, HIPAA, CCPA, biometric privacy, et cetera almost certainly will find claws into this behaviour.
Re: HIPAA, it would be the covered entity or business associate that is allowing xAI access to PHI who would be in violation, not xAI; same as Google isn’t responsible if someone sends PHI over Gmail and Google scans it.