I think by saying "you can't afford to build infrastructure where it's not crowded", he actually means that it's not affordable because the ridership will be too low, so it's not really viable in that sense.
I understand what he wanted to say, but it is same in the end - where it's crowded it is expensive to built and cheaper to operate, where it is not crowded it's vice versa, but in the end you end up with both being not affordable by his quote, so there is no difference between them really, they are both not financially efficient.