[adrtessier]

Hoo boy, most of these things don't worry me, but this one does.

I'm semi-responsible for some Juniper gear, thankfully all Junos (BSD) based, but I no longer trust any of it if this is malicious injection vs. a bad review. However, what the hell can I do? I can't audit the code. I trusted Juniper, and now I'm stuck with that trust being burned. Running to any other proprietary network vendor is just as uncertain.

If Junos gets a bulletin, I have a lot of work on my hands very soon, as do a good chunk of service providers. I remember there being rumors of a certain three-letter agency saying they had some type of exploit for the Cisco ASA as well; I wonder if it was something this deep, vs. just a run of the mill RCE vuln.

This is one more reason to use open-source products for actually security-sensitive systems, maintain a good amount of defense in depth, and do a little bit of auditing of the code you're using yourself. More often than not these days, it sure pays to be paranoid.

EDIT: At the same time, this also really makes me respect Juniper more than I have previously. A company that finds this internally, on their own audit, could have patched it silently and said nothing about it to anybody. It probably would have been better for them PR-wise. The honesty is worth me not jumping ship to another (probably compromised) proprietary vendor, but you betcha if I can get away with it, I'll run something open-source and community audited when I can.

[lmm]

I've always regarded networking equipment as outside my security boundary. All it does is forward packets to the right places; an attacker can deny service by shutting that down or sending them to the wrong place, but nothing else. All my connections are encrypted and authenticated at a higher level.

Do you terminate SSL or something on yours? Or have open unauthenticated services running on your internal network? If not, what's the actual threat here?

[technion]

- Plenty of services that can't adequately be secured any other way are often mitigated by restricting access to VPN users coming in over a network device. I appreciate that "just secure the service" is supposed to be the best practice, but when you're talking about things like IPMI interfaces or SCADA devices the alternatives approache zero

- Controlling the networking equipment can open you up to things like sslstrip

[rsync]

"things like IPMI interfaces or SCADA devices the alternatives approache zero"

My strategy with IPMI has been to assign IPMI non-routable, private IP addresses, then block that address space at the interior of the network (which is sort of redundant) and then require folks to SSH onto an interior host and connect to IPMI that way.

I would be very interested in, and receptive to, criticisms of this model ...

[lmm]

The argument there would be that it's very hard to secure all access to a network - anyone compromising any network device, or e.g. physical access to the cable runs in the building, then had access to IPMI.

In a high security situation I'd keep the IPMI network physically segregated, with a small number of machines acting as access to it. Or maybe connect IPMI only within each (locked) rack, and require using something like ansible if you want to perform an operation across more than one rack. Whether the cost/benefit fits for your circumstances is another question of course.

[technion]

If you consider "ssh jump host" and "vpn" somewhat similar implementations of the same general strategy (forcing users to jump through something secure) we have a similar recommendation.

[nathwhal]

In my experience, IPMI is generally considered a part of the control plane, and not accessed via the same network as apps/data, but through a separate, more restricted/audited privatenetwork.

[ghshephard]

Lots of enterprises use JunOS to terminate VPNs.

[ra1n85]

Nit: ScreenOS is the impacted network operating system, not JunOS.

[hueving]

Very few places follow that threat model. The cost of encrypting between the web tier and the db tier (even from a management perspective) is more than most organizations are willing to pay.

Your threat model is also missing the fact that a network can mitm your connections and also silently duplicate sensitive traffic.

[superuser2]

Well, no, not if it's SSL across the router. Only if it's terminating there or earlier.

[anExcitedBeast]

On an individual scale, you're right. But the fact is almost every corporation, non-profit, and government agency is weak inside the perimeter security stack; once you're in, you're in. A backdoor in networking equipment is a pretty serious problem.

[EthanHeilman]

What software do you use to achieve end-to-end VPN?

[jlgaddis]

If JunOS gets a bulletin, the whole Internet has a lot of work on its hands.

If JunOS gets a bulletin, shit, that's really, really bad.

I feel it's very likely that the NSA had something to do with the recent Cisco ROMMON "discoveries", so it would not surprise me one iota if they were involved here as well (although it's obviously pretty early to speculate on something like that -- and impossible to {dis}prove).

[adrtessier]

I am eagerly awaiting the incident report on this, although I find it unlikely we'll ever hear anything more than this from JTAC and friends. If this is the work of an intelligence agency, it will likely be gagged under the guise of "national security" unless there's a press-worthy indictment coming out of it.

[rodgerd]

> some type of exploit for the Cisco ASA as well

Given ASAs run a 2.6 kernel that's not hard. From my Kiwicon 8 notes on Alec Stuart-Muirk's talk:

* Literally every protocol handler has CVEs against it.

* Every time Cisco add a new one it gets at least a DOS CVE. (There are some proofs of concept for pivoting these into real exploits on other Cisco products.)

* The ASA’s high availability protocols are unauthenticated and unencrypted. This is bad. Like, “will accept any packet claiming to be a management packet as valid” bad.

* Some authentication is optionally available, but if you enable it, the ASA will still accept unauthenticated protocols.

[BuildTheRobots]

I this this might be the slides [1] -though if anyone has a video of the talk I'd love to watch it.

[1] https://ruxcon.org.au/assets/2014/slides/Breaking%20Bricks%2...

[jlgaddis]

Because they can (allegedly) survive software upgrades (on the ASAs and IOS routers), I've always believed that these "infections" are done at a lower level than the OS, such as in the ROMMON on the IOS routers.

After hearing about "SYNful Knock" recently, I'm inclined to believe this even more.

[x5n1]

Software based networking on BSD. Can't trust American vendors for anything. Is this what you wanted Mr. NSA? Good job at sabotaging your own business interests. It's not that you're spying, it's that you're so promiscuous about it.

[adrtessier]

Don't sell JunOS short. It is far more complex than "software networking on BSD" and has a lot of proprietary bits.

Junos (FreeBSD) is the Routing Engine; Juniper hardware also contains an ASIC-based Packet Forwarding Engine, which loads microcode from the Routing Engine upon boot. Not everything's in Junos all the time, but since the PFE loads its embedded OS from the Routing Engine kernel, you could just pwn the Routing Engine and then also have some sense of persistence in the PFE on reboot, probably. I don't know much about how the PFEs work internally.

I'm certainly no FreeBSD/JunOS expert. I am an unabashed fanboy of JunOS's *nix-y structure, though, vs. the monolithic binary that is IOS. (There was a great Blackhat 2011 talk on IOS reverse engineering, if you are interested in that sort of thing. [1])

[1] [PDF Warning] https://media.blackhat.com/bh-eu-11/Sebastian_Muniz/BlackHat...

[qb45]

I think your parent didn't mean that "JunOS is just software networking on BSD" but that "software networking on BSD is all that can be trusted because NSA screws commercial products".

And of course it doesn't have to be NSA. Maybe some foreign spies or pretty much anybody interested in spying on some Juniper's customers.

Or even a bored employee doing it for bragging rights. FWIW, I once worked for a (reasonably big) corp making software which has to run as root and I'm pretty sure I'd have been able to slip some small privilege escalation backdoor in there if I felt like doing so. But I have to admit that their products weren't as security critical (and, actually, already had some vulns), so one could hope that Juniper and Cisco are better than that.

[dingo_bat]

Have a look at http://openswitch.net/. Full disclosure: I work at HPE, though not in the Openswitch team.

[tw04]

That just means the exploits are pushed to the firmware blobs. Sure, it's some level better than something like JunOS, but just barely.

What's the point of locking your front door and leaving the window open?

[kijiki]

They don't even need to go after the firmware: Openswitch has a binary-only userspace process that talks to the hardware. That is where a bad actor would hide something nefarious.

[dingo_bat]

But you can take the OS and write your own driver for your own hardware. Also, you can take the OS as-is, run it on a VM, and examine the packets coming out for signs of any telemetry or any such nefariousness. I think it is a big step ahead of any other switch/router OS.

[kijiki]

None of the current forwarding ASIC vendors publish enough information to write a driver for their chips. So no, you can't write your own driver. The reason OpenSwitch's OpenNSL binary is binary-only is because it uses Broadcom's proprietary SDK, and programs registers that are only available under Broadcom's NDA.

If you're just going to run it in a VM, you'd be better off with more established things like OpenBSD, OpenWRT, pfSense, Cumulus VX, or even just a Debian VM with Quagga.

<disclosure>I'm CTO of Cumulus, whose network OS is exactly as open-source as OpenSwitch: everything but the single user-space program that links to Broadcom's SDK.</disclosure>

[dingo_bat]

>None of the current forwarding ASIC vendors publish enough information to write a driver for their chips. So no, you can't write your own driver.

Of course they don't publish it as open source. But if you buy their chips, they will, I assume. So then you can write your own driver.

>If you're just going to run it in a VM, you'd be better off with more established things like OpenBSD, OpenWRT, pfSense, Cumulus VX, or even just a Debian VM with Quagga.

I agree. My VM scenario was meant only to illustrate that OpenSwitch is really easy to run in a container/VM, and that fact can be used to examine it for any malovalent behavior.

[jlgaddis]

That's not really any different than putting a tap on both sides of one of these NetScreens, for example, and looking at the traffic. It's still impossible to "verify" a device isn't compromised that way.

Only when the public can view (ALL) the code, rebuild it, and install it on their own hardware can we be reasonably confident it has not been tampered with.

[chei0aiV]

Here is a Linux (Debian) based and more open similar thing:

https://cumulusnetworks.com/

[darkr]

It's not really much more open. Similar to some other network hardware (e.g extreme networks) it runs linux for the management interface. The switching/routing ASICs that does the actual work are still proprietary. It allows for a bit more flexibility in that it supports apt, but IMHO an image-based update system is preferable on network hardware, as you are not vulnerable to apt-breakage or anything like that. You know, whatever happens it will boot, always.