[stargrazer]

I will have to take the other side of that golden rule. Not sure where it came from. But when one has a decent handle on the tools at hand, they work wonderously well.

I have bare metal servers tied together with L3 routing via Free Range Routing running BGP/VxLAN. It Just Works.

No hard coded vlans between physical machines. Just point-point L3 links. Vlans are tortuous between machines as a Layer 2 protocol, given spanning tree and all of its slow to converge madness.

Therefore a different Golden Rule:

Always. Overlay. Your. Network.

Leave a note if you'd like more details.

[utopian3]

OP was mostly talking about cloud + docket containers. Your use-case is unrelated and seems to make sense.. But I still agree with OP and I believe overlays in the cloud is generally an anti-pattern of unnecessary complexity.