[motohagiography]

Let's see if 15+ years of security people getting after critical infrastructure asset owners like this has made any difference. At least they detected something and shut it down to control the response. They also know the costs to repair and replace things. I don't suspect the pipeline uses a federation of heterogeneous systems to operate its SCADA actuators, so I would speculate it is likely a single firmware vulnerability facilitating it.

The global chip shortage for replacement parts if they are needed seems like a strategic coincidence. Definitely an evolving story.

[procarch2019]

I work in control systems OT space. A lot of distributed control systems and scada systems interface with the business layer in some fashion to provide access to time series and event data and to allow for alerts via email/mobile. Some people do this properly with good network segmentation, firewalls, A/V and patching, etc (there are several standards that dictate best practice). That said, even when doing it properly you're introducing attack vectors. I don't think it would be a firmware vulnerability, but instead something malicious affecting the computers they use to control the process.

[motohagiography]

The reason I'm going for firmware is while the HMIs could have had a solarwinds style exposure, but that's just any generically wormable OS vulnerability, and not something that should cause a physical shutdown.

To shutdown a pipeline, it's not a management console issue, hence why I'd speculate it's in the ICS devices themselves, which probably use uClinux toolchains on SoCs from one or two large vendors. I did some smart meter and ICS security work in the 00's, and there were a few vendors who would be strategic targets. The attack tools available now are unbelievably better, while the attack surface is pretty much the same due to the long lifecycles of ICS components, and considering today we've got cheap SDRs and gnuradio blocks for most wireless protocols, AVR tools, buspirate and the good/greatfet, ghidra/ida, and python for reverse engineering, the vulnerability research on this stuff moves way faster than the industry ability to respond.

If this is a serious attack, the only way to respond will be if they are very lucky, it's a worm and they can stand up a honeynet with spare gear to catch a sample and any good infosec firm can pull it apart. But if it's an active APT group, there's probably a political solution, as given what's possible, this would seem to be just a shot over the bow.

[procarch2019]

I get what you're saying and that could very well be the case, but I think the 'pipeline' as a whole requires a lot of handshaking between the different stations. They would not be able to do this without their supervisory control later (or at least it would be particularly difficult). That alone could have caused them to shut it down.

Additionally, if there was a whiff of malicious software or unintended access I would imagine they would want to make sure it didn't get into other systems. That would involve isolating and possibly shutting down machines and equipment.

I guess we'll see when they release more information. I would imagine that we'll get more details since this is critical infrastructure.

[rhodozelia]

If the management console has a button or controls that would allow the person sitting at the management console to shut down the pipeline, which systems usually do have an emergency stop button in case there is an accident, then all you need is access to the management console to write one bit to the controller that says “operator pressed estop”

No need for firmware vulnerabilities in VxWorks when there are internet connected windows pcs.

[RhodoGSA]

Very interesting, kinda spooky.

Peer-to-peer threats from a world power perspective seem to be less bullets and more code. Any cyber warfare would just end in both parties destroying critical infrastructure until there's none left. War of attrition, skipping completely past the military and affecting the civilian population directly.

[sandworm101]

>> but instead something malicious affecting the computers they use to control the process.

I bet there is a layer of windows XP machines involved in a legacy control system. XP machines that weren't supposed to connect to the internet somehow have malware on them. It doesn't even have to do anything. Simply the detection of anything in such circumstances is enough to warrant them being shut down.

[procarch2019]

Totally agree, see it all the time. I even know of a few NT systems floating around out there. At least most companies are getting their IT involved to mitigate (usually they work with the vendor because they know nothing about control systems). They usually provide funding to the automation groups. People are starting to take it seriously.

[tw04]

Why wouldn't you use a unidirectional connection for time series and event data? I understand why you might want to send things out to the rest of the world, I can't fathom why you wouldn't require physical access to have write access.

[procarch2019]

Some time series data interfaces only work with tcp comms, which means you can’t always rely on unidirectional networks. I agree you should use them where possible though.

I replied to a comment on a dupe post regarding PAT, in which analysis is done on process data and fed back into the process to increase efficiency or yield. Obviously there are varying levels of criticality where the risk vs the business reward might not be worth it though.

[exikyut]

Genuine question (that I've been seriously wondering about for a long time): how do you implement validated attestation that a piece of log data has reached nonvolatile storage, triggered appropriate alarms, and that those alarm events have been acknowledged, while using a data diode type setup?

[mikewarot]

If it is critical to have the log, it has to be local. Infrastructure shouldn't die if an internet connection goes down.

You can sent the status of the log out through the data diode, along with a copy of the data.

[8note]

What do you do when this attestation fails? Eg. A fox chewed through the cable and the ack can't be received.

[jtchang]

Depends on your setup but a message bus architecture with polling would work.

[bilbo0s]

This.

I've said it a thousand times, all the security in the world will not defend a SCADA system if someone left TeamViewer running somewhere.

Don't mean to pick on TeamViewer. It could be any number of packages, but I think security minded people get an idea of the type of attack vectors I'm talking about.

[procarch2019]

It is mind boggling the lack of basic security principles some people have. I won't just put that on the plants and their IT/OT, or lack thereof. I've seen plenty of vendors and integrators do some cringe worthy stuff too.

[rhodozelia]

The whole automation industry is a security disaster but it is because security isn’t part of the deliverables for any party. It isn’t in the specs, civil, mechanical, electrical engineers it isn’t their responsibility.

If the owner has an IT department they usually don’t want to be responsible for it either since locking things down leads to weird issues with legacy proprietary SCADA systems.

There is no out of the box secure solution available yet. Rockwell certainly makes an attempt with their factory talk directory but I highly doubt that isn’t easily worked around somehow.

[procarch2019]

Yea, that is correct. I typically put together the solutions for new systems, including security. I give the sales team part numbers and hours for security software and related hardware. They then add that as an option to quotes. No principal automation engineer wants to take that on and no IT want to be involved. Also, when money is tight that’s an easy target for them to pass on.

Luckily I’ve pushed enough over the years that we at least include A/V software as mandatory.

I’ve been able to carve out a nice space within my company bridging the IT/OT divide. It’s been particularly good recently since the bigger companies are dictating good cyber practices, but rely on integrators and vendors to implement.

I don’t think there will ever be an out of the box solution unless a system stands on its own, which is becoming increasingly harder with modernization and reliability efforts. Add on top of that privileged access, remote monitoring and support, automated (kind of) patching, etc. you have to interface with the IT side a bit.

[rhodozelia]

Sadly the OT networks are 100% trusting of any device on the network. With Schneider plcs any device on the OT network can write to any addressed memory register over modbus - it’s like direct memory access DMA.

I hope that one day every device on the OT network has a yubikey and all messages are signed so that no unauthenticated access is possible.

[no-s]

> It is mind boggling the lack of basic security principles some people have.

OpSec - it's not just a buzzword, it's the Way.

[User23]

Shutting down pipelines is insanely expensive. Under normal circumstances maintenance work, including welding, is done on live pipelines. The guys that do that job are extremely well compensated, last I knew hundreds an hour, and maybe a little crazy.

A shutdown is a huge deal and means they’re taking this extremely seriously.