It matters if you let it matter. If you feel you need to go the whole Angel->VC schmoozing route, then definitely location matters. But there's a lot of other ways to get started.
It matters if your business is the type that can't prosper on determination alone. That kept us going for 6 years. Eventually we needed money and a supportive environment.