I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.
So, for clarity, it's never one fiber line being cut. It's a big massive bundle of fiber carrying different circuits for different carriers different places that were run in the same trench and all got cut at once.
Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one.
Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.
And when it's a large bundle cut, it's going to take a bit before service is restored. I have been told that a highly skilled tech can splice a single strand in a matter of minutes. Looking it up, I find figures of an average of 30 minutes per joint for 4-strand bundles, and slowing down from there. So if you have a 144-strand bundle to repair, or multiple 144-strand bundles, it's going to take several hours to effect the repair, and you're probably going to have to rotate through techs to avoid fatigue-induced errors slowing things down even more.
It is also, as a matter of practicality, going to be very difficult to find 144 colors that can‘t be mistaken for one another, particularly in conditions that i would imagine wouldn‘t have great lighting.
If they’re all in use and it’s point to point, might be easier connecting whatever to whatever. Then simultaneously swap line cards around as they get lit up or letting IP do its thing.
Not as feasible as it sounds initially. I don't know if you've ever looked inside a cat-5/6/whatever networking cable[1], but if you do, you'll see that it is actually a bundle of 8 individual wires, in 4 twisted-together pairs, color coded. Those different wires carry different electrical signals, and if the various signals are not supposed to be on the various wires they are supposed to be on, the connection won't work.
So imagine if you cut a Cat6 cable in half, and then spliced the individual wires back together without regard to which wires you connected. Very likely, Pair 1 negative is connected Pair 3 negative, Pair 2 has its polarity reversed, etc, and so the network link never comes online.
At the individual device level, fiber cables are connected in pairs, in a crossover fashion. Device A Tx connects to Device B Rx, and vice versa. So with a random splicing, you would end up with pairs getting crossed or broken up, invalidating the connections and labeling of every patch panel downstream, making for yet more work than the initial work of splicing the cut bundle.
1. Not impugning your intelligence, knowledge, or character here, some people never have because they've never needed or wanted to, and that's totally okay.
This. Even if you have redundant lines with multiple providers there is a good chance they are part of the same fiber bundle. It can be really difficult to get providers to admit who they are leasing from and what the physical runs are.
While I'm also not qualified to diagnose the problem, I would imagine it's something like this...
The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic.
The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
The odd thing about this situation though is the geographic location of Brooklyn: on an island. Why would a significant amount of traffic from outside Long Island be going across the harbor to Brooklyn, especially given all the major switching facilities in Manhattan (closer to shore).
What makes you think it is only connected to Manhattan? Long Island has submarine cables connected to New England, various points along the East Coast south of NYC, and several transatlantic cables.
My naive explanation: fiber networks handle incredible loads, and a single fiber line might correspond to 10+ Tbs (terabytes per second). So all that data needs to go somewhere else, and if it gets shunted, packets get backed up, resulting in a cascading failure.
Note: I am not a fiber technician nor do I work for a large ISP.
If a 100 Gbps link goes down, and all the other links that could carry that route are only 10 Gbps... it goes downhill quickly.
Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
Just to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!
Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one.
Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.