Yes, i mean what happens more often- if sign is not fully removed, but bent by collision or turned sideways by hooligans or something. Human would recognize that. Car will not.
Sounds like the cars should share a centralized database of critical sign locations, and cross reference their GPS location with the database. When they approach a location that the database says should have a sign, and one is not detected, it should behave as if there was a sign, and report the incident to somebody.