|
|
|
|
|
|
by sgc
1368 days ago
|
|
|
That is completely outside all my expectations prior to reading it. The consequences are potentially life and death, or incarceration, etc, and yet they did nothing until called out and basically forced to act. A good reminder that the bug can be anywhere, and when things stop working we often need to get very dumb, and just methodically troubleshoot. |
|
|
I used to install T1 lines a long time ago. One day we had a customer that complained that their T1 was dropping every afternoon. We ran tests on the line for extended periods of time trying to troubleshoot the problem. Every test passed. Not a single bit error no matter what test pattern we used.
We monitored it while they used it and saw not a single error, except for when the line completely dropped. We replaced the NIU card, no change.
Customer then hit us with, "it looks like it only happens when Jim VNCs to our remote server".
Obviously a userland program (VNC) could not possibly cause our NIU to reboot, right?? It's several layers "up the stack" from the physical equipment sending the DS1 signal over the copper.
But that's what it was. We reliably triggered the issue by running VNC on their network. We ended up changing the NIU and corresponding CO card to a different manufacturer (from Adtran to Soneplex I think?) to fix the issue. I wish I had had time to really dig into that one, because obviously other customers used VNC with no issues. Adtran was our typical setup. Nothing else was weird about this standard T1 install. But somehow the combination of our equipment, their networking gear, and that program on that workstation caused the local loop equipment to lose its mind.
This number-swapping story hit me the same way. We would all expect a compression bug to manifest as blurry text, or weird artifacts. We would never suspect a clean substitution of a meaningful symbol in what is "just a raster image".