Nah, I've been running in Five-fingers for years on pavement. As long as you take your time to ease into it so your body has time to adapt, you'll be fine.
Seconded. I've covered nearly 4,000 miles largely on tarmac/pavements in the last 2 years in Vibrams alone. I suffered with numerous running injuries in trainers before I switched. I've not had any running type injuries since I made the switch. They may not be for everyone, but they've certainly worked for me. Additionally I'm only on my 2nd pair in that time, although they are now slightly overdue for replacement.
This. Take your time. Baby steps. My wife and I use Vibrams as camp shoes for backpacking and I regularly wear them to the gym (treadmill) or around town. They've also helped me train to walk barefoot which I'll regularly do on lunch walks on the sidewalk. I've never had an injury in nearly a decade of using finger shoes casually.