"Couldn't afford" is a loaded term. I can afford a helicopter ride home, but I don't value its utility enough to pay what's being asked. Isn't that how prices are supposed to work?
Only if you equate price and value, but given vast discrepancies in wealth it doesn't really make sense to do that, since an extra $20 to someone making $250K/year doesn't have the same meaning as one making $25K/year. Consider the old way of doing such things, where there was a queue to get to the taxis, but everyone paid the same price. In that scenario the best measure of value is probably how long someone is willing to wait in line. Now imagine that same queue except anyone willing to pay twice the normal cab fare could just cut in line. Would you call that "allocative efficiency"? Or perhaps you would simply call it unfair?
For the queue, let's say Alice and Bob both make $25k/year. Alice desperately wants to get home to, I don't know, say goodnight to her kid and wake up early to get to work in the morning, while Bob is just going out drinking and doesn't really care about it that much. Is it efficient to prioritize Alice and Bob equally, or would you simply call it unfair?
In a surge pricing regime, Alice might be willing to pay, say, $50 while Bob is only willing to pay $10, and Alice will get a higher priority because she cares much more and her stuff is much more important. Isn't that a win?
I see where you're coming from with income disparities, but I think that talking about how it translates into problems with surge pricing is basically discussing the symptom rather than the cause. The root problem there is that one person can make 10x as much as another person without being anywhere near 10x more valuable. This causes a ton of secondary problems, of which the availability of taxis and taxi-like entities is one of the least important.