Though I've pointed out before that I don't think formal testing is a big part of Apple culture, I'm at a loss to come up with how this passed the most basic smoke testing. Or to flip it around, what random variable wasn't accounted for that allowed the devices in the test matrix to pass? Sure, it seems obvious that Apple just didn't test it. But c'mon, we know that can't be true. They must have loaded on some iPhone 6s. So what's the missing piece?
The update works fine if updated via iTunes. Only OTA updates are broken. Maybe they didn't test the OTA update process? It sounds absurd but it's the only viable option in my mind.
It's surprising to me that those would even differ in the first place. You'd think that you'd have a single deliverable which is delivered and signed, and the installer does the same thing regardless of where it gets the package. What makes OTA different from iTunes?
The obvious one would be carriers, especially since the error affects the phone part.
I have a hard time imagining a "basic smoke test" when it comes to a phone that has to actually try to connect to an external network, different depending on locations and contracts.
I agree, carriers might be the likely candidate. As for a smoke test, I'm seeing reports from users on US carriers. If it were "phones break on some obscure carrier in China", okay, fine. But when it appears to be breaking on carriers that are available to the testing labs right there in Cupertino, meh, it's got to be at least one other thing.
As another commenter said, I'd love to see a post morteum strictly in the interests of "don't let this happen to you". Sadly, we're unlikely to ever see it.
AFAIK the issue only affected OTA updates. I suspect that the vast majority of testing occurred without using OTA. I wouldn't be surprised if they are a bit more aggressive about testing OTA updates too after this.
So the root cause of this is similar to that of the broken Bumbeebee upgrade script [1,2]: a typo (missing quotation marks or unintended space) caused far more data than intended (a whole volume or /usr) to be rm'd.