[dododo]

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability.

e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

[rdl]

That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site).

Same with Juniper. (there aren't really other viable options besides those two)

You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense.

You end up with much worse reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.

[windexh8er]

Re: "there aren't really any viable options..."

Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with.

People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.

[rdl]

"there aren't really any viable options" to juniper or cisco for core/edge routers.

There are some routing protocols which interoperate (which is how different sites on the Internet can talk to each other), but most of the protocols used for HA or management of a given set of routers, or, more importantly, most tested/debugged implementations of HA and device management, are Cisco or Juniper specific.

No big deal announcing routes to your upstream if you use Juniper and they use Cisco. Big deal if you have Cisco+Juniper and want to do HSRP (Cisco-only).

[windexh8er]

Well, no.

I've been in network engineering for 12+ years and I fundamentally disagree with a lot of what is said about "networking" and interop by many programmer-types (not casting here, but) on HN. Yes, yes, you may understand system DevOps to a point, however I'm not sure you've spent a significant amount of time studying Dijkstra's algorithm or truly have an idea of how to deploy a global IPv6 overlay. I'm also not trying to be snide here but I feel that, often times, many things that come up on HN are just fundamentally designed wrong from PHY all the way up until the devs get a hold of the rest. I've been in a very successful startup (think one of the top online backup services) wherein their network was run on commodity junk hardware. They were asking me how I'd troubleshoot this, that and the other thing - obviously with no debug (this guy said that with a grin). First and foremost, you designed it wrong - I can show you inefficiency in about 10 minutes of performance engineering that I would have designed around without thinking about those things. So, yes, I can waste time tracking down a bad NIC on your network, but if you feel that you've earned geek cred because you fired up Wireshark and parsed through a few simplistic ARP tables - you haven't impressed anyone but yourself. That's when I realized I was working with professional developers, and not network architects.

Your simplistic view of FHRP is trivial at best. Maybe if you were talking about how you'd design fault tolerance into a virtual link, say an LSP, with something like BFD in your design I'd be more impressed than conversations about proprietary redundancy protocols of which most network engineers won't touch for a variety of other reasons than the big "C".

</endrant>

[rdl]

Virtually no network engineers (by percentage) have to do anything other than worry about what their vendor supports for a given configuration (and usually a fairly small set of configurations, too); it's much more about policy and operations.

Similarly very few developers have to solve open CS problems in writing a CRUD application (or I guess more comparably to ops, come up with a novel implementation of a complex algorithm).

This is progress, though.

[windexh8er]

"Virtually no network engineers (by percentage) have to do anything other than worry about what their vendor supports for a given configuration" - this statement puts a perspective on your thinking. And then I read your information on the services your company offers, and I realize that it's not worth having a discussion.

"<redacted> takes your security very seriously." - right. That's a statement, not information regarding the thought or implementation. There's not even a mention of technology. <sigh>

[justincormack]

Thats why software defined networking, Openflow etc, are going to take off, as you can get back control of the protocols and what is going on, and avoid the vendor lockin.

[sneak]

> Thats why software defined networking, Openflow etc, are going to take off

I've been hearing this for a decade. It's still not true. I'm not sure why, either.

[rdl]

What do you call Arista? Also there is a lot of interesting "virtual appliance" networking going on.

I agree the right choice today is almost certainly a C or J router and probably C or A switches, but e.g. hardware load balancers like F5 seem to be losing out to software in most deployments (increasingly).

I built a decent sized network with Zebra 15y ago, which was pretty obviously the wrong tech, but interesting.

[FlukeATX]

Cisco + Juniper environment and you want gateway redundancy?

Hello, VRRP!

There are open standards for pretty much every Cisco proprietary protocol. Even EIGRP is now available as an informational RFC.

[pyvpx]

you can't utilize OSPF, ISIS, BGP, et. al. for high availability? You do realize HSRP is merely for a redundant gateway address, right?

most service provider networks manage via the CLI (generally scripted, for better or worse) and occasionally a vendor-specific API.

while I will agree juniper and cisco are generally the best choice for core/edge routers, there are other 'viable' options depending on your requirements. if you need in excess of 2500 BGP sessions on a single chassis, there aren't many viable options besides a Cisco 7600.

I certainly do not mean to be rude, but I feel you're attempting to speak from experience you don't fully have (yet, hopefully!)

[windexh8er]

Wow, just... Wow. So. Far. Off. Base. (see my post above)

[grogers]

I view it less as a cost factor and more as a convenience. Developing expertise with juniper and Cisco takes a lot longer. Each router vendor has its own quirks. Even just buolding software to monitor routers is basically a full time job since its a constantly moving target. New bugs are always coming up....

[scoot]

"But, the reality of the situation is that all of this gear is marked up well into the thousands of percent."

You seem to be confusing hardware with software. Juniper's gross margin was 64.25% for the quarter ending Dec. 31, 2012, and in that ballpark for previous quarters back to inception.

[packetslave]

You'd be amazed how often "standard" network protocols behave subtly different between vendors. You have to exhaustively test interoperability for every single feature and config option if you want assurance that it isn't going to break in some bizarre way.

[rdl]

It is also really nice to be able to call one TAC and have them devote effort to fixing it. If you have a heterogenous network, they can pass the buck, or even if they are awesome and try to help you out, there is no way Cisco's TAC knows as much about Juniper stuff as they do about Cisco, it is harder for them to put together a duplicate config, etc.

Back around 2000 this was a big deal. Cisco slacked on gigabit routers, and Juniper didn't have a comprehensive product portfolio, so while SP networks could be all J (but maybe with some switches from Extreme, etc,), enterprise networks were a lot more likely to have juniper and Cisco mixed, if they needed juniper performance in the core. Juniper ended up broadening their portfolio and Cisco improved their high performance offerings a few years later.

[DASD]

http://www.zdnet.com/uk-internet-hit-by-linx-router-failure-...

Here's an example of a large provider with parallel infrastructure each powered by a different hardware provider(Brocade/Extreme). One failed and one kept working. I seem to recall a more detailed RFO but my Google-fu has failed me this morning.

[rdl]

LINX just runs inter-provider switch fabric, though, which is vastly simpler, and just runs two separate switch fabrics for customers to plug into.

Running an anti-DDoS/CDN service which handles traffic like Cloudflare does would be vastly more difficult.

It's certainly possible to do, but I think the given ~reasonable engineering resources, the net reliability of a heterogenous J/C version of CloudFlare would be less, and performance worse, than what they have now.

Switch fabric is a lot closer to the "run different models of hard drives" (although, you don't do that WITHIN a RAID group either -- you do it on separate RAIDs and possibly separate chassis), than routing infrastructure (which is like running a 777 with 1 GE engine and 1 RR engine. At best, you can turn it back into a 747 and run 2 GE engines and 2 RR engines.)

[DASD]

I'm not going to go much further because this debate is useless without a context of limits and expectations. No one is discussing simplicity. LINX's operation is not simple. A specialized provider is offloading a difficult function as a core competency in return for simplicity. As an end-user, the difficulty is a non-factor. Just make it happen. Couple in "reasonable" with expectations and then we know what to expect. If it costs the moon to never make this happen again, then charge accordingly. If this happens once in a blue moon, then charge a lesser price.

[inopinatus]

I'm afraid this perspective is misconceived. I, too, used to believe it. After all, in any portfolio, risk is reduced by diversification, right?

Unfortunately, amortized over the lifetime of a computer system, risk is not reduced in this manner. There is no hedging of vendor vs vendor in a technical portfolio; what happens instead, for any tech of significance, is the internal development of an abstract control plane that can communicate with both, and that control plane is then the single source of defects. In the meantime your engineers have to become world-class experts on two platforms rather than one. In practice the divided loyalties will turn one world-class engineer into two half-assed ones.

Domino-effect failures, or global misconfiguration failures like those experienced by Cloudflare are edge cases in my experience and not something you should optimise for. When they happen they tend to be catastrophic, but worse is the insidious decline in quality caused by carrying too much technical debt.

Cloudflare's scenario is not comparable to the installation of a RAID set. They are more comparable to a developer of RAID controllers. The experience curve for such is very, very long.

Not saying they couldn't have done other things to make this situation less catastrophic, but diversity of core technology portfolio isn't a winning ticket.

[dageshi]

Will customers be willing to pay for the additional costs incurred by that inconvenience? I think there are a whole lot of different things to try before you start introducing different routers with different os's/quirks/capabilities into the mix. Frankly that sounds like a recipe for not just inconvenience but chaos.