[exelius]

I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out.

What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them.

And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?

[NoMoreNicksLeft]

The big trouble is that we need settlement-free peering, or we'll have a very different sort of internet, or maybe none at all. Though it's hard for some to remember (or to believe), there were big consumer networks prior to the advent of consumer-oriented Internet. Compuserve and AOL were their own networks. And they were abysmal. Peering not only ruined that business model, it managed to co-opt those networks and assimilate them into the Internet proper.

If peering is untenable because it leads to outcomes like the one we have now (Netflix) where it is fundamentally unfair to one party (assuming that is the case) or the other, then we're all screwed. There might not be a fair solution that still manages to resemble the Internet.

In one alternative, the Comcasts and the AT&Ts are smited and no longer exist, and Level 3 goes into the consumer internet business, connecting things end-to-end. That's not something we could trust them with. Same if Comcast replaces Level 3.

In another, Netflix continues to pay the extortion (if it is that) which we have little doubt that Comcast will continue to ratchet up the price for. Or Comcast has to shoulder that burden alone (and can't charge customers extra for doing so, without people screaming "net neutrality!").

Or maybe services like Netflix just can't exist in such an environment. Also a bad outcome.

Or god help us, internet infrastructure is nationalized, and the same people who manage our roads and traffic lights take over.

Am I blowing this out of proportion?

[toyg]

You could have a situation where the government mandates net neutrality and at the same time compensates the likes of Comcast in some way (recognising their role as public utility and basically sanctioning their monopoly, but without taking over their operations entirely). You could have an anti-monopolistic legal action that splits Comcast but prevents Level3 from taking over / being present in that market. Etc etc...

There are many ways to skin an onion, and full-scale nationalisation is probably the least likely outcome (cash-strapped governments don't need another headache right now). In fact, the real problem here is that one player is leveraging a monopolistic position; remove that position, and the Free Market should start working its magic again.

[paganel]

> time compensates the likes of Comcast in some way

Serious question, how would one find the exact sum with which Comcast should be compensated in the absence of a functional market?

[lmm]

Analyse the available facts and make a best guess, just like we do with private companies operating as regulated monopolies in e.g. water supply.

[zero_intp]

Comcast already enjoys a physical government granted monopoly of the 'cable' access rights to their customers.

[malchow]

How about: a representative Congress mandates strong net neutrality rules and Comcast runs a 20% margin business instead of a ~83% margin business?

[NoMoreNicksLeft]

We tried that once with Bell Telephone. Their profits were dialed down a bit via regulation, and it was an oppressive, horrible, and un-innovative network. You weren't allowed to buy a telephone and plug it into the jack at home... you could only rent them from the company.

Thankfully, that company and environment are gone. Let's not recreate them.

[notdonspaulding]

> it's a 20:1 disparity in traffic in vs traffic out.

Are you talking about on this graph?

http://blog.level3.com/wp-content/uploads/2014/05/route_info...

Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.

> ...transit providers are starting to see their industry be squeezed by the big ISPs.

No.

> ...why do transit providers even exist?

Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).

Take another look at the L3 network map from the blog post:

http://blog.level3.com/wp-content/uploads/2014/05/network_ma...

If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.

> Settlement-free peering assumes that bandwidth usage is roughly symmetric

No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.

Comcast empirically does not care about the goodwill of its customers, which is not surprising given its monopoly of local markets. If customers had alternatives to Comcast and it became known that Netflix worked on one ISP and not the other, customers would switch ISPs en masse.

[exelius]

I was going from memory on the 20:1 thing and I was wrong, but 5:1 is still not even close to symmetric.

I also used a bit of hyperbole in the "why do transit providers even exist" part; transit providers do still need to exist to provide access for the long tail. But Level3 and the like do see their market shrinking as the big fish who currently pay them for transit either directly or indirectly move more bandwidth to direct connections with the big ISPs. It is definitely starting to squeeze them.

As far as the benefit to each network goes, that's hogwash. It's not how the contracts are written because you can't quantify "benefit". You can quantify packets sent/received. Netflix needs to make money just as badly as Comcast does, so the fact that their service is crappy is much more their problem than it is Comcast's. Netflix can always purchase transit through another company that's not Comcast.

[zero_intp]

I can assure you that L3 and Cogent are not seeing the marketplace shrink. 10k quarterly files from each discuss their growths and vulnerabilities.

Historically, eyeball networks have paid for transit from backbone providers. As market consolidation occurred, the cable providers with government granted physical monopolies were able to negotiate for better settlements with their transit providers.

Eventually, the cable companies condensed with the major telephone companies. Some of the new mega companies have backbones and can use their own networks for transit (Verizon, AT&T, ...) others (Comcast) were able to use their size and access to their customers to negotiate largely settlement free exchanges.

However it is important to note that ALL eyeball networks have 1:5 to 1:20 demand ratios. This is the nature of content versus consumption. There is no new news regarding these ratios, and they are not particularly germane unless attempting to engineer the flows.

The real and pertinent issue is that Comcast has not lived up to the 'timely upgrade' clauses in boilerplate (NDA'd) settlement-free peering agreements. Clearly they have an advantage to "defect" from the standard cooperation model. Netflix has chosen to change providers several times, and recently provided data on whom they pay for service.

[nitrogen]

Why should an end-user ISP ever expect a settlement-free peering bandwidth graph to be symmetric? The important question is whether customers are requesting connections that route through through a particular transit provider. Comcast gets to sell "Internet" access, while the transit provider gets to sell bandwidth. Win win, no symmetry needed.

...you can't quantify "benefit"....

Isn't that the entire purpose of an economy?

[wdewind]

> To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.

Can you qualify this a bit? What's the actual expect cost to Comcast here to fix the issue? General ballpark?

[notdonspaulding]

You're right to call me out here. Although I have a reasonable idea what it would take technically, I don't actually know the financial impact to Comcast for increasing capacity to L3 or Cogent (not even ballpark). However, I'm taking the L3 article at face value when it says:

    But there are also typically shared costs for networks to 
    interconnect. Each party pays to augment its own network 
    to allow for more traffic exchange (the expense to augment 
    capacity is not significant for either party). And since 
    we often choose to interconnect in a third party data 
    center, the networks usually agree to share the cost of 
    the cross connects by paying for them on an alternating 
    basis.

I take that to mean the CapEx is insignificant and the recurring expenses are shared equitably. What's notable about that is that Comcast isn't even sitting down at the table to negotiate equitable terms for upgraded capacity. They're out and out refusing to upgrade without direct payment.

[zero_intp]

Cogent's CEO has already offered to pay outright for the port costs and data-center cross-connects for any upgrades. The point is to prove that the infrastructure cost of the interconnect are not the issue.

[PeterisP]

Upgrading that interconnect mentioned in the original article seems like just plugging in some $10k-$20k standard hardware and connecting a single extra wire. That kind cost doesn't matter really, it seems that they're hurting their interconnects intentionally for political reasons.

[johnjac]

This "out vs in" comparison in order to determine who pays is a meme that won't die. I've never known agreements to be formed in such a way. Maybe before my day in the dial up era when "out vs in" could serve as rule of thumb of sorts to determin who was an access ISP and who was an "Upstream" or "Tier 1" ISP. Can someone provide a source for this?

[wmf]

Historically, ratios were used for peering between ISPs of the same type. When one backbone peers with another backbone, 1:1 ratio can be a simple proxy for equal value. (ISPs liked simplicity in the old days because it made the Internet cheaper than the complex telco X.25/ATM/SONET networks with their complex tariffs and settlement.) When a content provider peers with a broadband ISP, they can exchange equal value without exchanging equal traffic.

[dllthomas]

"Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay."

I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.

[exelius]

Because the cost of a packet is borne by the receiver while the sender is the one monetizing it.

The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.

[njharman]

> you are being paid for that activity through subscription fees, ads, donations, etc.

Last time I checked ISPs where heavily monetizing packet receiving to the tune of 5-10x what netflix monetizes from me. For typically slow, shitty service bundled with cable TV or phone I do not want.

[dllthomas]

... and it's probably pure coincidence that Netflix is in direct competition with 1/3 of that bundle, too.

[wmf]

Because the cost of a packet is borne by the receiver...

That's true for hot-potato routing, but CDNs generally use cold-potato routing.

As for monetization, both Netflix and Comcast are getting paid by their respective customers.

[exelius]

It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute.

What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).

[nitrogen]

Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them.

Customers pay Comcast with money to establish those links. If that's not "monetizing", what on earth is?

[namecast]

The original L3 article actually touches on this, sort of, by mentioning markets where ISP port congestion is not an issue (i.e. the UK) due to competition between ISPs.

Comcast will get paid by their customers as long as the routes they pass traffic through are 'good enough'. They have a (near) monopoly on the last mile in many markets, so 'good enough' can mean both 'barely working' and 'better than any other option you've got' at the same time.

So, you're right, but so is the parent: Comcast has no financial incentive to provide any more than just enough bandwidth to keep you from calling the support line to complain or cancelling your contract in frustration.

Where's the rest of customer revenue going, if not towards network upgrades? Well, let's just say the Comcast Center (http://archrecord.construction.com/projects/bts/archives/off...) ain't paying for itself...

[dllthomas]

As others have said, Comcast is also being monetarily rewarded here. I think the best argument lies along "the sender is in the best position to take measures to reduce traffic (traded off against their goals)."

[exelius]

They are rewarded for providing access; not for passing individual packets. There's a difference.

[dllthomas]

How precisely are they providing access if they do not pass individual packets?

Edited to add: It's certainly true they are not rewarded for the marginal packet, but then neither is Netflix (given their subscription based model).

[zero_intp]

Level-3 is actually going around to their peers and trying to create systems to measure the bit-miles traversed on traffic sent to other tier-1 peers; the carriers will then pay for out-of-sync ratios

[wpietri]

> Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay.

Does it assume that?

Since about the time of the web browser, we've had consumer-focused ISPs. Consumer traffic is mostly small requests for large replies. But as the article suggests, Level3 has a lot of settlement-free peering with consumer ISPs. My belief is that being settlement free isn't about equality of packets, but equality of demand. E.g., I have a lot of servers made to serve consumers; you have a lot of consumers wanting access to my servers; let's just split the costs.

In any case, given that those Comcast packets are mostly requesting all those Level3 packets, it seems much fairer to charge Comcast disproportionately, as its their customers who are creating the demand.

[nullc]

Right, there is a disparity — so, explain why it shouldn't instead be Comcast paying Level3?

Even if you want to privileged "sending" as being more charging worthy since you control what you send— in this case it's Comcast customers that have requested those bits, hosts connected to the level3 network are providing them.

Networks like comcast have built out imbalanced networks which likely wouldn't exist without their access to monopoly infrastructure. Practically anyone they peer with except other consumer broadband monopolists is going to be unbalanced.

[venomsnake]

Torrenting provided a lot of traffic that could bring symmetry to the amounts send/received. And yet it was frowned upon.

[jessaustin]

Indeed: perhaps the answer to the symmetry argument is for Netflix clients to just start uploading scads of random data while the movie is streaming. Then the traffic will be "symmetric", although Comcast won't like it after all the last-mile upload links are saturated and customers can't do anything online.

[exelius]

Netflix hasn't done this (and likely won't) because would make it difficult/impossible to control the quality of their service. P2P connections are a bit of a crap-shoot based on things like router configuration, device type and resources. For example, P2P wouldn't work as well on an iPad where you're only caching a few hundred megs of video (and not the whole video) at any given time.

[dllthomas]

I don't think the parent meant to imply a P2P implementation of Netflix streaming video, just that Netflix is perfectly capable of making that ratio balance out by uploading crap from the clients they have (partial) control over - which would then be traffic coming from Comcast's network.

[jessaustin]

Thank you, I was having trouble imagining how I could have been that unclear.

[nullc]

Torrenting moves large amounts of traffic into last mile infrastructure which tends to already be congested because its very costly to upgrade.

Due to using many distinct flows it also tends to not be very TCP friendly and takes a fairly unequal share of the bandwidth. And while users will give up and do something else when their interactive service becomes slow, a congested torrenting host keeps torrenting.

[dublinben]

There doesn't seem to be any congestion on the last mile infrastructure of providers like Comcast. Their problems are all with interconnection to the real internet. It would actually be better for Comcast if their customers were getting more content from eachother over p2p networks than from external providers like Netflix.