I've never observed street light color making driving at night annoying; for the 3-5 second interval the yellow is on.
Just curious, have you ever been tested for color blindness?