As of February 2016, there were 1530 CHAdeMO high-powered chargers in the US, compared to only 253 Tesla Supercharger stations. Tesla is falling behind.
You're right quantity-wise, but then you have https://twitter.com/elonmusk/status/812708946225963008 . I'd be surprised if Tesla EV's don't soon charge faster than it takes to fill up gas, within a couple years even. If EV's in general take off though, I think it's true that Tesla may have trouble keeping up with the larger market in terms of chargers.