You can hash them without a salt and store them in a set of passwords not associated to user accounts to enforce uniqueness without having to actually know the passwords
That still introduces a fairly serious vulnerability. The lack of salting on the "password uniqueness" database makes it a juicy target; an attacker with access to the database can attack those passwords, then try the ones which are known to be valid from there against the salted passwords in the user database.
I wonder if there’s some way to mitigate this by either only keeping the uniqueness database long enough to identify duplicates and then deleting it or by using this on lower priority systems that people may reuse passwords from for your higher security one. In either case, the small number of bad passwords you would identify that you couldn’t come up with yourself or find on common password lists probably makes this a bad tradeoff.