For local networking, packet loss is, in most cases, a sign something somewhere isn't doing well. So, maybe, it's better notice it sooner than bump into possible unexpected problems later.
But those buffers are also often huge and your packet could still come out the other end of one of them 20ms (or 2s, if it's a cable or DSL modem) later. You still want to treat it as evidence of congestion, though.
Extremely long tail latencies are observed which reach a second or more in those networks when some interference causes loss.
It would be much better to react quicker than the eternity of 200ms (if that is the default).