[anmorgan]

> Software is so complex that nobody really understands it.

Unless this statement is trying to imply something deeper than what it states, I would disagree.

I work with embedded systems and connected devices. To me it's not a problem of complexity, but a problem of not having a strong standard of practices on both the device side and the facility side.

Have sensitive data on your network? Separate it from the rest of the network, or don't put devices on the network that don't meet your security needs.

[jeffdavis]

"don't put devices on the network that don't meet your security needs"

Does it not disturb you that the mere presence of a device on your network could compromise everything else? Why is that?

I argue it's because of complexity. We don't really know how these devices behave outside of very controlled circumstances.

And the vulnerability could be anywhere in the stack. I remember this bug from a while back:

https://lcamtuf.blogspot.com/2014/10/psa-dont-run-strings-on...

The gnu strings utility was vulnerable to untrusted input! Who would possibly imagine that would ever be the case!

Honestly, that bug just made me give up. Software cannot be reasoned about any longer. I used to believe that solid components strung carefully together could add up to something understandable. But no, we are beyond that.

I'm not talking about crazy James Bond hackers that somehow infected your compiler or something. I mean that, by accident, a basic utility does something crazy.

This is not math or science or engineering any more. It's wizardry, witchcraft, and alchemy.

(I'm exaggerating a bit, but it really is discouraging to me.)

[blub]

You might find Normal Accidents by Perrow interesting. He describes accidents caused by systems (e.g. nuclear power plants) becoming so complex that they are incomprehensible to humans.

"I used to believe that solid components strung carefully together could add up to something understandable. But no, we are beyond that."

I think that's true, but it turns out that we don't have any solid components. None.

[jeffdavis]

If you put two nuclear power plants on the same grid, it is pretty hard to imagine how a meltdown of one plant would trigger meltdowns elsewhere on the grid (because the grid carries electrical energy and is incapable of carrying high-speed neutrons). But with software, you don't have to imagine such failures, they happen all the time (because the internet can carry any data, including more software). So I still maintain that software systems are more complex. And if they aren't more complex today, they will be soon, because the complexity is growing without any obvious bound.

But for the sake of argument, let's say they are of comparable complexity. If you show a layperson a nuclear power plant, and say "who do you think should run this: you, or a team of nuclear engineers?" they would probably answer "a team of nuclear engineers, please". Show the same person a television, and they will feel like they should be able to operate it. But it's actually an internet-enabled TV running sophisticated software that is on the same home network as your internet-enabled security camera system, and it's a very unsettling situation. In other words, now software makes everything -- toasters, TVs, phones, cars -- into incomprehensible systems.