[rhino369]

Like the tango, it takes two to interconnect. It's a bit biased to say ISPs fail to provide broadband speeds. You could easily say L3 or Cogent fails to provide it's subscribers the transit service they are paying for.

ISPs are rightly a little pissed that Cogent, for example, takes Netflix's money for transit service then hot potatoes the data right onto the ISP for the ISP to transit to the end customer.

Remember, Cogent is being paid to deliver the content, but they are contracting to carry more data that the interconnect can hold. Now you could say the ISP should just let them expand the interconnect, but that is no excuse for Cogent overselling their own capacity.

Interconnects have always been a wild west.

[msandford]

You could argue that, but Cogent is willing to provide the interconnect to its customer, Netflix. But what we're seeing is that the ISP that is selling internet to its customer isn't willing to provide sufficient bandwidth to meet their expected-but-not-contractually-obliged obligations.

They can do this because they have a monopoly on their customers. Not just a "Microsoft monopoly" where there are plenty of alternatives but network effects that keep people using Windows, but honest-to-god monopolies not just granted but ENFORCED by the government.

What we have here is a failure of the government to break the monopolies, revoke the monopolies, or police the monopolies. A little legislation that said "you can't oversell your bandwidth by more than X amount" would go a long way towards giving consumers an effective stick to beat their ISP with for over-promising* and under-delivering.

Note: The * is there to denote that they've written the contracts such that basically no matter what happens they're not in violation of them. We're only entering those contracts willingly in the sense that it's an abusive contracts from one of two or three vendors, or no contract at all. It's kind of like asking someone if they'd rather be beaten or stabbed. Given the choice; neither. If I have to pick one, how big is the knife?

[rhino369]

1) The ISPs are willing to interconnect, they just aren't willing to interconnect in an unbalanced fashion without being compensated. Settlement free peering is free because the two sides transfer the same amount so it's a net of roughly zero. In these cases, the net is heavily on the Cogent or L3 side.

Now I think there is a decent argument that ISPs should have to have free peering available at their local loop level. If congent wants to peer at the CO office of the local telecom, it should be able to do it.

The issue is that Verizon is not just a local ISP, it's a nationwide T1 provider. It sells transit service. It should be allowed to charge for transit to it's own local loop networks.

2) There are no enforced monopolies anymore, and haven't been for a long time. Damn near all profitable areas have a telecom and a cable provider. Congress banned monopoly franchise agreements for cable and fiber in 1992. Cable Television Consumer Protection and Competition Act of 1992, Pub. L. No. 102-385, 106 Stat. 1460 (1992).

3) I believe ISPs should have to meet some benchmarks for speed compared what they advertise, but you can't force them to maintain speed on specific interconnects. If Cogent has a shitty network that isn't on Verizon.

[AnthonyMouse]

> Now I think there is a decent argument that ISPs should have to have free peering available at their local loop level. If congent wants to peer at the CO office of the local telecom, it should be able to do it.

That's what the whole dispute is about. "Verizon is a Tier 1 provider" is some kind of meaningless talking point with no relevance to anything. Cogent and Level 3 are already transit providers with national networks, they don't need to use Verizon's transit network because they have their own. The problem is that they have traffic they want to pass to Verizon in LA for Verizon's LA customers and Verizon won't give them access without paying a toll. The fact that Verizon also has fiber going from LA to Houston and San Francisco has no effect on any part of that.

[AnthonyMouse]

Actually, let me revise that. The fact that Verizon is a Tier 1 provider does have an effect -- it's the motive for Verizon's anti-competitive behavior.

The traditional arrangement is that you have a last mile provider in each region and a collection of Tier 1 providers who they pay to provide transit between them. If a last mile provider is also a Tier 1 network then the relationship is different. Rather than a monopoly last mile network shopping for a transit provider and the transit providers all competing with and peering with each other, you have a large monopoly last mile provider who doesn't need a transit provider and is in competition with the other transit providers.

The situation changes from one where Standard Oil is buying trucks from Ford and Chevy to one where Standard Oil starts making their own trucks and can charge prohibitively high prices for fuel to Ford and Chevy customers to destroy their business and force customers to buy the monopolist's trucks.

[rhino369]

No it's not what the dispute is about. The Verizon/Cogent or L3 peering points are nowhere near the local loop.

Verizon isn't a fake T1, it's a real honest to goodness T1.

[AnthonyMouse]

> The Verizon/Cogent or L3 peering points are nowhere near the local loop.

Do you have some kind of cite for this? If it didn't already matter which Verizon peering point they were using then it wouldn't matter if any particular one was congested because Cogent could just dump the traffic on a different one in a different city.

Is your point that Level 3 isn't a Tier 2 network but Verizon is? Obviously Level 3 is going to peer with Verizon somewhere in Greater Los Angeles and Verizon would carry the traffic from there to Long Beach or whatever. But then you would seem to be suggesting that the Tier 1 network should be paying the Tier 2 network.

> Verizon isn't a fake T1, it's a real honest to goodness T1.

The Tier 1 network they own is not fictional, it is only not relevant. The traffic for Verizon's customers in Los Angeles is not delivered to Verizon in New York.

[dublinben]

Have residential ISPs selling heavily asymmetrical connections (25:1 ratios aren't uncommon) ever had roughly equal in/out traffic with Tier 1 providers?

Haven't they been the party to buy transit in the past?

Aren't Comcast and Verizon just taking advantage of their size and monopoly position to play hardball with the Tier 1 providers?

[rhino369]

1) Probably not but it is much more one sided with media streaming. This is why I said that ISPs should be forced to peer for free at the local loop level. But outside of that, they are acting as a transit network.

2) Small ISPs buy transit if they need it. Comcast and Verizon don't because their larger networks acts as a transit for them.

3) Yes they are taking advantage of their size, but they are performing the same service as a transit provider. If Verizon spun off their local ISP network, Cogent would have to pay VerizonT1 to transit to the local ISP.

Why does Verizon owning the transit and local somehow make it unreasonable to ask for payment?

Verizon and Comcast aren't asking for money to accept data, they are asking for money to transit the data for Cogent and L3.

[lacksconfidence]

Why would Cogent still have to pay VerizonT1 to transit to the local ISP if it was spun off? Cogent have their own nation wide network to transit with.

[No1]

>1) The ISPs are willing to interconnect, they just aren't willing to interconnect in an unbalanced fashion without being compensated.

Keep in mind the ISPs provide highly asymmetrical connections to their customers, prohibit those customers from running servers, then cry foul over an imbalance.

Their consumer networks are designed to be imbalanced; there is not even a chance of them being balanced.

> ...but you can't force them to maintain speed on specific interconnects. If Cogent has a shitty network that isn't on Verizon.

No one is proposing holding Verizon accountable for Cogent's performance across Cogent nodes. It is possible to mandate packet delay / rejection limits specifically at the interconnect.

[msandford]

> Keep in mind the ISPs provide highly asymmetrical connections to their customers, prohibit those customers from running servers, then cry foul over an imbalance.

Actually, it's worse than that. They then cry foul over the people who are doing their best to provide that asymmetric data without complaining. Why aren't Level3 and Cogent bitching about the imbalance?

From a very high level the internet can (and should) be thought of as pull-only. If you start sending me data that I have't requested, most people would consider that a DOS. If a bunch of people start sending me data I haven't requested, that's a DDoS. If both parties consent to the data being transferred, that's GENERALLY done through a pull-style arrangement.

Yes I do realize that uploading files to a website might be considered and colloquially referred to as "push" but if the machine that's receiving the upload stops sending ACKs then the machine sending the upload stops sending more data.

The ISPs are selling as asymmetric service but then getting upset if you use it asymmetrically because they assumed that the average utilization of the service might only be 0.1%. In reality it's turning out that the average utilization is more like 1% or 5% and this means they have to upgrade their infrastructure continually rather than one-and-done. I'm not sure that I feel badly for them. It's obvious that people use a lot of bandwidth and they're only going to want to use more.

[jauer]

Open peering at common points on the part of the T1 providers would help.

If you're a small ISP or datacenter (T2/T3) Cogent/L3/etc won't peer with you even if traffic levels are balanced. They'd rather you have to purchase more transit instead because then there's a chance you'll purchase transit from them. OFC now they gripe about MSO/ILECs not peering with them. As a small T2 (peering + transit) provider, cry me a river. They are getting what they deserve.

Part of the problem is Cogent (and their ilk) primarily ride cheap intercity fiber between NFL/MBA cities and ignore the other places. Fiber to COs is the province of the ILEC, MSO, or boutique transport provider (INDATEL etc).

If it doesn't make sense for Cogent/L3 to pay the ILEC/MSO for ports to offload traffic at a datacenter it certainly doesn't make sense for Cogent to buy fiber to the CO from ILEC.

[click170]

1) This point is silly because it is made moot by Netflix configuring the client to upload as fast as possible whenever they're watching a movie, which I think we can agree would benefit nobody.

Yes, it's true that this is how peering agreements have worked in the past, but if that's not sustainable going forward maybe another agreement needs to be reached.

2) What you say may be true, but it doesn't change the fact that nearly every market in the US has but one single ISP provider. That sure feels a lot like a monopoly to me.

3) I largely agree with you here, except that I would clarify that you can force them to maintain speed between the interconnect. Both the interconnect and the ISP have to have capacity to spare, you can't under-provision your link to the interconnect and then point the finger back at them.

[FireBeyond]

You can't give your end users vastly asymmetrical data pipes (cable/DSL), and then complain that you don't need to equitably handle traffic for a service then because the data flow isn't symmetrical. At the very least, it's disingenuous. You -know- your traffic flow to the greater internet is asymmetric but you want to bitch that other flows to you are asymmetric. Do as I say, not as I do.

[rhino369]

1) Netflix could try, but ISPs would be well within their rights to block Netflix attempting generate useless traffic on their network. Even net neutrality rules would allow that for that type of action.

They could shift to a sort of bittorrent distributed system if they wanted to, but that is a huge shift in models and not really worth talking about.

2) IIRC 79% have two providers. Their local telecom and cable company.

3) You can but it's a crappy policy that would destroy the ISPs negotiating position. Potentially even letting the transit providers charge the ISP, like they already charge small ISPs.

A) Remember, Cogent and L3 are still selling Netflix and all their other customers, space on interconnect links that it doesn't have.

In what other business can you sell your client service and then fail to deliver and just blame your supplier for not giving you a cheap deal.

B)If you are going to force free peering on ISPs they should at least be able to charge the peer the actual cost to set up the peer point.

[click170]

> In what other business can you sell your client service and then fail to deliver and just blame your supplier for not giving you a cheap deal.

Isn't this exactly what the problem ISPs are demanding, which is what we're up in arms about? They're trying to double dip by getting their customers to pay and trying to extort Netflix and Level3 as well.

At least we can both agree that this is wrong.

[msandford]

Precisely. Comcast sells me X megabits per second and then, if it's to Netflix, refuses to deliver it. Anywhere else? Sure, no problem! But Netflix? 2-3Mbps tops. And I'm paying for 50Mbps. And if I VPN to ANYWHERE, magically I can get full speed, full quality Netflix without any problem.

What Comcast is (and other ISPs are) doing is intentionally not spending the $10k-$50k that it would take to upgrade the peering with Cogent or Level3 so that they can deliver to me the bandwidth that they ostensibly have promised but which they are contractually not obliged to provide. And people think that's shitty, because it sure seems like a monopolist abusing their position of power. Perhaps because it is.

[pmb]

It certainly could be either side of the interconnect that's causing the problem. The important insight is that no ISP is uniformly bad at either end, which means that the problem is not at either end, but in the middle. So this study has found congestion in the core (at the interconnects), precisely where it is not supposed to be, and has found that congestion to be so bad that it drives performance to sub-broadband levels.

Cogent and L3 are ISPs as well, so I think saying "ISPs are failing to provide broadband speeds" is fair. However, to add context, L3 had a blog post a while back explaining that they are standing ready to add more capacity for cheap and that the congestion is, in at least one instance (in LA), Verizon's fault. http://blog.level3.com/open-internet/verizons-accidental-mea...