Most drivers ignore them, but it only takes 1 driver going at the 20mph limit in a bottleneck to slow down a whole column of traffic. So both are true.
OK, I'm not sure it's going to make that much difference since 20 mph zones tend to be in residential zones, typically rat runs. This indicates to me that high volumes of traffic are more likely the cause of congestion rather than some annoying people who insist on obeying the law.
Let's, for the sake of argument, say I am one of these annoying people. I will drive at 20 mph with a tail of 6 angry drivers behind me until I reach a junction or a set of lights that invariably already have cars waiting to move forward.
All I am doing is delaying the drivers behind me from reaching this queue of cars a few seconds later than they would otherwise.
Of course, their rage is directed in my direction but that's another story.