[Habgdnv]

Okay, looks like I will reply to a few of the comments to clarify things. I’ll give a concrete, real example.

I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).

That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.

Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.

The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.

Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)

[Hikikomori]

For your second scenario you should announce the /23 to both and each /24 to one of them. Usually you can also prepend your own AS, ISPs I've worked could also prepend for you with select communities.

I don't think your cases are good enough to allow anyone to spoof by default.

[Habgdnv]

I said that we tried this. They do not care about the announce. They care about what is injected into the routing table. We are announcing it but they see better path and drop us.

And they also said that it should work that way - just announce it somehow and it will work. Yes but no. It does not work.

[Stefan-H]

In your first scenario, any connections established through the ISP-A's IP address would be routed back through the VPN connection that they came in on. If that server were to establish it's own connections to external resources, it would feasibly be able to use the 10g connection from ISP-B. It would not be able to dictate what source address was used with connections coming from ISP-B.

[jcalvinowens]

It could work the way OP described if they routed all outbound traffic via ISP-A regardless of source address, and ISP-A allowed spoofing. I think that's what they meant.

[Habgdnv]

It is common practice for business subscribers (around UK) to get a /29 On the router we add a single /32 via the tunnel.

I think even the cheapest 100bucks business plans from many ISPs come with /28 or /29. It is a complete waste because we had like 10 offices with 3-5 persons with laptops and NO servers. The common question from the ISPs is: Do you need some IPs? When we answer no, they give us /29.