[delinka]

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router.

BGP is the process by which ranges of IPs are claimed by routers. Is Calico really used by docker containers in this way?

[atombender]

That is indeed the use case being solved here.

Kubernetes enforces a specific rule: Each pod (a group of containers) must be allocated its own cluster-routable IP address. This vastly simplifies Docker setups: In a way, it containerizes the network, just like Docker containerizes processes. It's the only sane way to manage containers, in my opinion.

This system requires something that can hand out IPs and ensure that they're routable on every machine. That something can be done in different ways, range from extremely simple to rather complex. For example, you could have something that acts like a bridge and coordinates with other nodes to find available IPs, and simply maintains the routing table on the nodes themselves in sync with this shared database (Flannel can run in this mode). Or you could use an SDN-defined overlay network (e.g. Weave).

[q3k]

This works as long as you're willing to stretch an L2 network to reach all the hosts of your container platform. That doesn't really scale.

Using a real routing protocol also immediately gives you access to traffic control, shaping, monitoring and redundancy tools, hardware support and knowledge that network administrators have been applying for years.

[bboreham]

People are networking their containers on IP address ranges segregated from their host addresses. So the routers have no idea where to send the packets, unless you take some additional steps.

[iheartmemcache]

Yeah, have an upvote. This is totally a case of "using an industrial core drill with tungsten carbide bits when all you need is an Ikea drill and a chinese cheese-grade bit".

Sounds like this guy just found out about a cool new tool and decided to blog about it. BGP can be used on local networks but it's total overkill for a docker situation where all of your instances are likely in the same rack (often on the same machine!). Ifyou don't even have an AS from ARIN/RIPE there's no reason to even touch this (as you pointed out, it's all the protocol is designed to broadcast to the public internet -- e.g. 'hey I own this AS which has rights to this net-block, direct packets in this fashion please!' Jeez.

I have no idea what the CPU overhead of running this is but I'm sure it's not trivial, especially if the BGP daemon is tuned to retain any significant amount of the whole BGP table (RAM/swap issues galore I'd imagine). Granted the article is titled 'one way'.. which is empirically true, it's a Rube Goldberg way of going about networking.

(n.b., OSPF is what people use for "BGP" within your own intranet, even when you have tens of thousands of boxes. It's called "border gate protocol" for a reason..)

(Sorry, I don't use docker so I can't actually make a constructive comment telling you what the canonical/right way of doing it is, but I can assure you, this is not it.)

[jauer]

(I didn't downvote you, but this might be why someone did...)

Eh, BGP on local networks is common enough that it was novel maybe a decade ago. It's perfect for running on servers to announce /32 addresses upstream to ToR switches. OSPF is actually more heavyweight & complex since you have to carry link state, do neighbor elections, etc. Ref: NANOG talk in 2003: https://www.nanog.org/meetings/nanog29/presentations/miller....

You don't even need an AS from your RIR for BGP to be useful on internal networks, just pick one (or more) of the private ASNs and roll with it.

Current best practice for internal networks (on the SP side at least) is to use OSPF to bootstrap BGP by enabling IP reachability for p2p & loopback addresses. After that customer / service routes are carried by BGP, not by OSPF. This is because BGP doesn't spam link state and has better policy knobs. You get a double win because OSPF converges faster with fewer routes and if you have redundant links your BGP sessions won't flap because the loopback addresses stay reachable. Ref: NANOG talk in 2011: https://www.nanog.org/meetings/nanog53/presentations/Sunday/...

CPU/RAM overhead is insignificant with a bgpd like BIRD or Quagga. They work for internet-scale routing (currently over 610,000 prefixes) with trivial load. An Atom C-series with a few GB of RAM can deal with internet BGP routing updates (CPU becomes significant for packet forwarding, not so much maintaining routing tables).

I'll take a boring 10-year-old routing setup using iBGP on cluster of route reflectors running BIRD with servers numbered out of a /24 per rack routed by a ToR l3switch with each service using BGP to announce a /32 over new and exciting L2 overlay networks any day. Troubleshooting is easier without having to work through different layers of encapsulation, dealing with MTU, trusting a novel control plane for whatever overlay network your using, etc.

[gonzo]

> CPU/RAM overhead is insignificant with a bgpd like BIRD or Quagga. They work for internet-scale routing (currently over 610,000 prefixes) with trivial load. An Atom C-series with a few GB of RAM can deal with internet BGP routing updates (CPU becomes significant for packet forwarding, not so much maintaining routing tables).

We've got a DPDK-enabled VRouter that will run over 12Mpps on a C2758 (8 core) with a full BGP table.

[X-Istence]

> (Sorry, I don't use docker so I can't actually make a constructive comment telling you what the canonical/right way of doing it is, but I can assure you, this is not it.)

And you are categorically wrong. This is a very good way to do it.