It gets worse, some drivers seem to adopt an adversarial attitude towards the usage of turn signals and will deliberately accelerate or do something else to make things more difficult to you.
If two cars travelling the same direction contact each other such that the front axle of one car bumps the rear axle of the other (the car in front), it's usually the car in front that has the safety problem. This is the same principle behind the PIT maneuver.
Otherwise (changing lanes, left turn across traffic), I think it’s more “not ahead of ME!” than about the signal itself.