I agree, but driving while high is not at all safe. Sure, it's less bad than drinking, but it's still more dangerous than driving when sober (and rested). Accident rates have gone up since legalization. [0,1,2]
It depends on what you measure. Crashes are going up ever so slightly vs population growth, but fatalities per crash has astoundingly been reduced by about 70%, which is to say that you were more than 3x as likely to die in crash in 1980 vs 2011.
That's because we have better cars now which offer better protection for passengers and pedestrians: seat belts, air bags, "softer" bumpers, collision detection, etc.