The additional 165 dollars to get a passport for the first time is quite steep for a document that seems to become more and more mandatory. Papier, bitte.
Countries with national IDs charge you to replace one if it gets lost, and it usually costs less than 10 USD.
I'm also not quite sure how you get a passport without a bank account. Can you pay in cash? Even if you can, I'm guessing that's only at certain offices, which adds to the hassle and cost.
What people are arguing against is that making having a passport mandatory to participate in society is an unreasonable burden, under the current structures. If you wanted to mail a passport to every American in the mail, at no cost, no questions asked, that would be a very different proposition to what is being discussed.
You can't guarantee every citizen has a passport, so if you were running this as a bank or an employer or so on an API that only took passport information would not be super helpful. When I think of a unified ID I think of a number everyone gets at birth tied to an ID card they can show you. Social security is closest to this but the cards say they're not supposed to be used for identification and it's a cludge.
Countries with national IDs charge you to replace one if it gets lost, and it usually costs less than 10 USD.