[pmb]

Original title was something like: US ISPs are failing to provide broadband speeds during peak hours.

If I could make it longer, I might have said "Due to neglect of relatively cheap interconnects, US ISPs are and were measured to be providing sub-broadband connections to many portions of the Internet".

The title "ISP Interconnection and its Impact on Consumer Internet Performance" just says that there was a study and says nothing about its results, but it's the results that are interesting. It was kind of a surprise to see the title changed out from underneath me. I did not know HN titles were edited in that way.

[rhino369]

Like the tango, it takes two to interconnect. It's a bit biased to say ISPs fail to provide broadband speeds. You could easily say L3 or Cogent fails to provide it's subscribers the transit service they are paying for.

ISPs are rightly a little pissed that Cogent, for example, takes Netflix's money for transit service then hot potatoes the data right onto the ISP for the ISP to transit to the end customer.

Remember, Cogent is being paid to deliver the content, but they are contracting to carry more data that the interconnect can hold. Now you could say the ISP should just let them expand the interconnect, but that is no excuse for Cogent overselling their own capacity.

Interconnects have always been a wild west.

[msandford]

You could argue that, but Cogent is willing to provide the interconnect to its customer, Netflix. But what we're seeing is that the ISP that is selling internet to its customer isn't willing to provide sufficient bandwidth to meet their expected-but-not-contractually-obliged obligations.

They can do this because they have a monopoly on their customers. Not just a "Microsoft monopoly" where there are plenty of alternatives but network effects that keep people using Windows, but honest-to-god monopolies not just granted but ENFORCED by the government.

What we have here is a failure of the government to break the monopolies, revoke the monopolies, or police the monopolies. A little legislation that said "you can't oversell your bandwidth by more than X amount" would go a long way towards giving consumers an effective stick to beat their ISP with for over-promising* and under-delivering.

Note: The * is there to denote that they've written the contracts such that basically no matter what happens they're not in violation of them. We're only entering those contracts willingly in the sense that it's an abusive contracts from one of two or three vendors, or no contract at all. It's kind of like asking someone if they'd rather be beaten or stabbed. Given the choice; neither. If I have to pick one, how big is the knife?

[rhino369]

1) The ISPs are willing to interconnect, they just aren't willing to interconnect in an unbalanced fashion without being compensated. Settlement free peering is free because the two sides transfer the same amount so it's a net of roughly zero. In these cases, the net is heavily on the Cogent or L3 side.

Now I think there is a decent argument that ISPs should have to have free peering available at their local loop level. If congent wants to peer at the CO office of the local telecom, it should be able to do it.

The issue is that Verizon is not just a local ISP, it's a nationwide T1 provider. It sells transit service. It should be allowed to charge for transit to it's own local loop networks.

2) There are no enforced monopolies anymore, and haven't been for a long time. Damn near all profitable areas have a telecom and a cable provider. Congress banned monopoly franchise agreements for cable and fiber in 1992. Cable Television Consumer Protection and Competition Act of 1992, Pub. L. No. 102-385, 106 Stat. 1460 (1992).

3) I believe ISPs should have to meet some benchmarks for speed compared what they advertise, but you can't force them to maintain speed on specific interconnects. If Cogent has a shitty network that isn't on Verizon.

[AnthonyMouse]

> Now I think there is a decent argument that ISPs should have to have free peering available at their local loop level. If congent wants to peer at the CO office of the local telecom, it should be able to do it.

That's what the whole dispute is about. "Verizon is a Tier 1 provider" is some kind of meaningless talking point with no relevance to anything. Cogent and Level 3 are already transit providers with national networks, they don't need to use Verizon's transit network because they have their own. The problem is that they have traffic they want to pass to Verizon in LA for Verizon's LA customers and Verizon won't give them access without paying a toll. The fact that Verizon also has fiber going from LA to Houston and San Francisco has no effect on any part of that.

[AnthonyMouse]

Actually, let me revise that. The fact that Verizon is a Tier 1 provider does have an effect -- it's the motive for Verizon's anti-competitive behavior.

The traditional arrangement is that you have a last mile provider in each region and a collection of Tier 1 providers who they pay to provide transit between them. If a last mile provider is also a Tier 1 network then the relationship is different. Rather than a monopoly last mile network shopping for a transit provider and the transit providers all competing with and peering with each other, you have a large monopoly last mile provider who doesn't need a transit provider and is in competition with the other transit providers.

The situation changes from one where Standard Oil is buying trucks from Ford and Chevy to one where Standard Oil starts making their own trucks and can charge prohibitively high prices for fuel to Ford and Chevy customers to destroy their business and force customers to buy the monopolist's trucks.

[rhino369]

No it's not what the dispute is about. The Verizon/Cogent or L3 peering points are nowhere near the local loop.

Verizon isn't a fake T1, it's a real honest to goodness T1.

[AnthonyMouse]

> The Verizon/Cogent or L3 peering points are nowhere near the local loop.

Do you have some kind of cite for this? If it didn't already matter which Verizon peering point they were using then it wouldn't matter if any particular one was congested because Cogent could just dump the traffic on a different one in a different city.

Is your point that Level 3 isn't a Tier 2 network but Verizon is? Obviously Level 3 is going to peer with Verizon somewhere in Greater Los Angeles and Verizon would carry the traffic from there to Long Beach or whatever. But then you would seem to be suggesting that the Tier 1 network should be paying the Tier 2 network.

> Verizon isn't a fake T1, it's a real honest to goodness T1.

The Tier 1 network they own is not fictional, it is only not relevant. The traffic for Verizon's customers in Los Angeles is not delivered to Verizon in New York.

[dublinben]

Have residential ISPs selling heavily asymmetrical connections (25:1 ratios aren't uncommon) ever had roughly equal in/out traffic with Tier 1 providers?

Haven't they been the party to buy transit in the past?

Aren't Comcast and Verizon just taking advantage of their size and monopoly position to play hardball with the Tier 1 providers?

[rhino369]

1) Probably not but it is much more one sided with media streaming. This is why I said that ISPs should be forced to peer for free at the local loop level. But outside of that, they are acting as a transit network.

2) Small ISPs buy transit if they need it. Comcast and Verizon don't because their larger networks acts as a transit for them.

3) Yes they are taking advantage of their size, but they are performing the same service as a transit provider. If Verizon spun off their local ISP network, Cogent would have to pay VerizonT1 to transit to the local ISP.

Why does Verizon owning the transit and local somehow make it unreasonable to ask for payment?

Verizon and Comcast aren't asking for money to accept data, they are asking for money to transit the data for Cogent and L3.

[lacksconfidence]

Why would Cogent still have to pay VerizonT1 to transit to the local ISP if it was spun off? Cogent have their own nation wide network to transit with.

[No1]

>1) The ISPs are willing to interconnect, they just aren't willing to interconnect in an unbalanced fashion without being compensated.

Keep in mind the ISPs provide highly asymmetrical connections to their customers, prohibit those customers from running servers, then cry foul over an imbalance.

Their consumer networks are designed to be imbalanced; there is not even a chance of them being balanced.

> ...but you can't force them to maintain speed on specific interconnects. If Cogent has a shitty network that isn't on Verizon.

No one is proposing holding Verizon accountable for Cogent's performance across Cogent nodes. It is possible to mandate packet delay / rejection limits specifically at the interconnect.

[msandford]

> Keep in mind the ISPs provide highly asymmetrical connections to their customers, prohibit those customers from running servers, then cry foul over an imbalance.

Actually, it's worse than that. They then cry foul over the people who are doing their best to provide that asymmetric data without complaining. Why aren't Level3 and Cogent bitching about the imbalance?

From a very high level the internet can (and should) be thought of as pull-only. If you start sending me data that I have't requested, most people would consider that a DOS. If a bunch of people start sending me data I haven't requested, that's a DDoS. If both parties consent to the data being transferred, that's GENERALLY done through a pull-style arrangement.

Yes I do realize that uploading files to a website might be considered and colloquially referred to as "push" but if the machine that's receiving the upload stops sending ACKs then the machine sending the upload stops sending more data.

The ISPs are selling as asymmetric service but then getting upset if you use it asymmetrically because they assumed that the average utilization of the service might only be 0.1%. In reality it's turning out that the average utilization is more like 1% or 5% and this means they have to upgrade their infrastructure continually rather than one-and-done. I'm not sure that I feel badly for them. It's obvious that people use a lot of bandwidth and they're only going to want to use more.

[jauer]

Open peering at common points on the part of the T1 providers would help.

If you're a small ISP or datacenter (T2/T3) Cogent/L3/etc won't peer with you even if traffic levels are balanced. They'd rather you have to purchase more transit instead because then there's a chance you'll purchase transit from them. OFC now they gripe about MSO/ILECs not peering with them. As a small T2 (peering + transit) provider, cry me a river. They are getting what they deserve.

Part of the problem is Cogent (and their ilk) primarily ride cheap intercity fiber between NFL/MBA cities and ignore the other places. Fiber to COs is the province of the ILEC, MSO, or boutique transport provider (INDATEL etc).

If it doesn't make sense for Cogent/L3 to pay the ILEC/MSO for ports to offload traffic at a datacenter it certainly doesn't make sense for Cogent to buy fiber to the CO from ILEC.

[click170]

1) This point is silly because it is made moot by Netflix configuring the client to upload as fast as possible whenever they're watching a movie, which I think we can agree would benefit nobody.

Yes, it's true that this is how peering agreements have worked in the past, but if that's not sustainable going forward maybe another agreement needs to be reached.

2) What you say may be true, but it doesn't change the fact that nearly every market in the US has but one single ISP provider. That sure feels a lot like a monopoly to me.

3) I largely agree with you here, except that I would clarify that you can force them to maintain speed between the interconnect. Both the interconnect and the ISP have to have capacity to spare, you can't under-provision your link to the interconnect and then point the finger back at them.

[FireBeyond]

You can't give your end users vastly asymmetrical data pipes (cable/DSL), and then complain that you don't need to equitably handle traffic for a service then because the data flow isn't symmetrical. At the very least, it's disingenuous. You -know- your traffic flow to the greater internet is asymmetric but you want to bitch that other flows to you are asymmetric. Do as I say, not as I do.

[rhino369]

1) Netflix could try, but ISPs would be well within their rights to block Netflix attempting generate useless traffic on their network. Even net neutrality rules would allow that for that type of action.

They could shift to a sort of bittorrent distributed system if they wanted to, but that is a huge shift in models and not really worth talking about.

2) IIRC 79% have two providers. Their local telecom and cable company.

3) You can but it's a crappy policy that would destroy the ISPs negotiating position. Potentially even letting the transit providers charge the ISP, like they already charge small ISPs.

A) Remember, Cogent and L3 are still selling Netflix and all their other customers, space on interconnect links that it doesn't have.

In what other business can you sell your client service and then fail to deliver and just blame your supplier for not giving you a cheap deal.

B)If you are going to force free peering on ISPs they should at least be able to charge the peer the actual cost to set up the peer point.

[click170]

> In what other business can you sell your client service and then fail to deliver and just blame your supplier for not giving you a cheap deal.

Isn't this exactly what the problem ISPs are demanding, which is what we're up in arms about? They're trying to double dip by getting their customers to pay and trying to extort Netflix and Level3 as well.

At least we can both agree that this is wrong.

[pmb]

It certainly could be either side of the interconnect that's causing the problem. The important insight is that no ISP is uniformly bad at either end, which means that the problem is not at either end, but in the middle. So this study has found congestion in the core (at the interconnects), precisely where it is not supposed to be, and has found that congestion to be so bad that it drives performance to sub-broadband levels.

Cogent and L3 are ISPs as well, so I think saying "ISPs are failing to provide broadband speeds" is fair. However, to add context, L3 had a blog post a while back explaining that they are standing ready to add more capacity for cheap and that the congestion is, in at least one instance (in LA), Verizon's fault. http://blog.level3.com/open-internet/verizons-accidental-mea...

[JshWright]

How have you been here for more than half a decade and haven't noticed that title editing is favorite pastime of the mods here...?

Generally speaking though, the original title is the one to us, and if you use something else, it will generally be changed to match the original one.

[jrochkind1]

Except when they consider the original title "click baity" and change that too.

I've given up on trying to come up with what I think is the most useful title to HN though, or on trying to guess what the mods will do. I just use the original title and let the mods change it if they want. If I think the original title isn't good at explaining the main point of interest from a HN view, I feel sad, but oh well, that's clearly how HN wants it.

It might be convenient if HN had a tool like reddit to automatically fetch the title from HTML given a URL. In fact, if they had such a tool, I'd expect HN to maybe actually mandatorily make that the title and not allow you to edit it, since it seems to be HN editorial preferences not to let submitters write titles.

[dang]

> Except when they consider the original title "click baity" and change that too.

The HN guidelines (edit: which are at https://news.ycombinator.com/newsguidelines.html and linked at the bottom of every page) say not to use the original title when it is misleading or linkbait. How could that be clearer?

Do we judge every case correctly? Good lord no. That's why we change things when people point out our mistakes. The community contribution here is considerable, so the best way to complain about a title is to suggest a better one.

The guideline also explains why titles can't be automated in the way you suggest: the software would have to identify what is misleading or linkbait. If anyone can show us such software, believe me, no one would use it more eagerly.

[jrochkind1]

My opinion on what is linkbait or misleading is often not the same as yours. But that's fine, that's why you're the mods.

My titles have been changed so many times (in both directions), that I just decided to always use the original title, and let you do your mod job and change it according to your judgement.

Do the HN guidelines also say to always use the original title unless it's misleading or linkbait too? I hadn't actually been aware of that -- where do I find these guidelines? I'm not actually sure. If I had known that, it would have spared my energy in coming with titles I thought were more informative only to have them, as a rule, changed back.

Thanks for your work!

I just found the Hacker News Guidelines you mention by googling -- previously I hadn't been aware they even existed, so didn't know to look for them. I see they do say to ordinarily prefer the original title. I am not sure how people find out about these guidelines, are they linked to from somewhere on HN that I'm missing? It might make a lot of sense to include a link to them on the 'submit' form.

[wmf]

The guidelines are linked on the bottom of every single page of HN (edit: except the submission page itself!) but I guess that's the part that nobody reads. Maybe they should be more prominent in the submission flow.

[dang]

> It might make a lot of sense to include a link to them on the 'submit' form.

That's a good idea. It's easy for us to forget that not everyone knows about this, and I'm sorry for assuming that you did.

> let you do your mod job and change it according to your judgement

I hope you didn't miss my point above about how we get it wrong too. The most I'll claim is that we try to correct the mistakes people tell us about.

[sillysaurus3]

I've given up on trying to come up with what I think is the most useful title to HN though, or on trying to guess what the mods will do. I just use the original title and let the mods change it if they want. If I think the original title isn't good at explaining the main point of interest from a HN view, I feel sad, but oh well, that's clearly how HN wants it.

I'm glad HN is like this. It's one of the reasons I enjoy the website.

It might be convenient if HN had a tool like reddit to automatically fetch the title from HTML given a URL. In fact, if they had such a tool, I'd expect HN to maybe actually mandatorily make that the title and not allow you to edit it, since it seems to be HN editorial preferences not to let submitters write titles.

This was tried, I think. It was proven to be a bad idea in practice. If you think the complaints about titles were bad now, you should've seen them when the software did automated reversions. At least, I think they were automated reversions.

Sometimes the subtitle is a better title. Sometimes the original title is clickbaity, as you said. Regardless, it takes a person to moderate the titles.

[dang]

Begging your pardon, that wasn't the original title. You editorialized it by picking a detail you consider important and interpreting it. That breaks the HN guidelines. The policy here is to let readers make up their own minds what's important. Indeed, the first reply disputes your interpretation.