[zbisch]

I'm sorry to sound very negative on the author, but I don't know how you could have the knowledge to write this article, but not have the knowledge to understand the basics of costs on the Internet.

A few points that are horribly wrong FTA (reasoning explained below):

"In order to use as few network resources as possible—both to keep costs low and quality high—these providers typically attempt to get as “close” as possible to end users, often forging agreements in which they directly connect to provider networks."

The "close"-ness that they are talking about here is not geographic closeness but network closeness. And this isn't just in terms of the distance traveled geographically, but the cost to send that traffic across some other network. That is, it is more expensive for Netflix to send traffic from their provider to (for example) AT&T to Sprint to Comcast to you, than it is to have Level3 send traffic from Level3 to Comcast to you. In fact, because of the peering agreement between Level3 and Comcast, Comcast has to pay Level3 to send them traffic (or so I believe).

The cost is not how many links it has to traverse, but the cost of the peering agreement that that network has with where it is sending the traffic.

"The only appreciable differences between the Xfinity streaming service for Xbox and e.g., Netflix, are that the source of content is within the Comcast’s “internal” CDN instead of on a third-party CDN, and that Comcast requires you to be using their own Internet service. (This is much more likely related to their agreements with content owners rather than any technical reason.) "

This is wrong. The reason is entirely financial. It is cheaper for Comcast to send traffic only within their network. If I traveled to China, it would cost Comcast much more to send the traffic to China and almost nothing to send the traffic to my computer which is Comcast's network.

"As you’ll see, the cap-exempt content is likely even more expensive for Comcast to deliver than the third party content!"

"All of these third-party streams almost certainly originate from third party providers in the Bay Area, all via direct connections to Comcast. Even though they count against my bandwidth cap, they almost certainly traverse fewer fiber route-miles and physical router ports (Comcast’s two primary costs of delivery) than the stream which originated in Seattle(!) and does not count against my cap."

Completely wrong.. Even though it is a further distance away, sending traffic from Seattle to somewhere in California is almost no cost to Comcast, so long as it stays within their network. With retrieving content from Level3 (such as Netflix movies), it has to _pay_ Level3 to send them traffic that its users want.

Again, sorry to bash on the author so much, but the author was misinformed concerning cost. Also, DSCP is kind of meaningless when traveling between networks (any network router can overwrite them and change their priority).

[wmf]

It has been reported that Level 3, Akamai, and Limelight are paying Comcast and delivering content as close to the user as possible, so I agree with the original author that external content would seem to be cheaper than Comcast's internal CDN (maybe even a profit center).

http://arstechnica.com/tech-policy/news/2011/02/peers-or-not...

http://apps.fcc.gov/ecfs/document/view?id=7021030795

[zbisch]

Ah, very good point. The last thing I read on the Level3/Comcast spat story lead me to believe Comcast was paying Level3 for the peering. I apologize. However, this line lead me to believe the author had a misunderstanding of the costs of traffic:

"...they almost certainly traverse fewer fiber route-miles and physical router ports (Comcast’s two primary costs of delivery)."

That is, unless the author meant Comcast's primary costs within their network, which is not what I read it as. Without providing reference to the cost of the peering, the comparing the _geographic_ distance of traffic seemed irrelevant. Even in this case, the reason that it is more expensive for Comcast to send traffic is not that is coming from Seattle, but because they are losing money on having Level3 pay them.

Edit: I also need to read more on this as I'm curious why Level3 is paying Comcast since Comcast is (as far as I know) not acting as a transit network. But I suppose this is what happens when you violate valley-free routing.

Edit2: Also, just wanted to add, the fact that Comcast controls the peering links is why this is so sketchy. By raising the costs of peering links, they could make it way more expensive for the competitors to send their users traffic, forcing the competitors to raise their prices, allowing Comcast to be the only content provider to have reasonable prices. While the part about the bandwidth to them not counting towards your cap is sketchy, the practice of charging you less, a lower flat rate, or not at all for traffic that stays within the network is reasonable. Where it is sketchy is when that difference is leveraged to make the service cheaper. If the service is more expensive, it's the fact they are charging competitors more that is sketchy. So... in general, providing content and the network is all around sketchy.

[bjornsing]

Just a quick comment on cost (since OP and these comments seem to have it a bit backwards):

The real cost in a DOCSIS network like Comcast's is actually in the last mile. DOCSIS is sort of like a mobile network in a pipe. The base station in this case is called a Cable Modem Termination System (CMTS) and data is transmitted from there over RF with QAM modulation to all the cable modems on the same coaxial segment. This can be hundreds or even thousands of cable modems. For unicast IP the packet is only received by one of them but they all have to listen.

When the radio spectrum inside the coaxial cable becomes congested the segment has to be split in two parts and a new CMTS installed, very similar to how you split a mobile network cell into sectors to improve capacity. Like with a mobile network this is capital intensive.

Not saying this makes Comcast right. But important to understand where they're coming from.

[zbisch]

Residential networks are actually part of my current research, and, although I don't know the full details of every technology, I am familiar with the basics of how DSL and DOCSIS networks work.

Yes, there is a cost (in terms of performance and congestion) over these links, but this not really relevant in the discussion when talking about the cost of where the traffic is originating. Whether the traffic originates in Comcast's data center in Seattle, from a CDN in Level3's network, or over their peering links with Tata, the cost of it traversing the last-mile is irrelevant. In any of these cases, traffic will still be crossing this link. And yes, this is a significant cost in terms of deploying the last-mile network, but as I said, it has nothing to do with the cost of getting of getting the packets to the DOCSIS network.

What we're saying is that if Comcast is carrying traffic from Seattle to a user in their network, they do not have to pay a transit provider to carry this traffic. If it is from Level 3, Level 3 would actually be paying them to carry that traffic to Comcast subscriber. If the traffic were coming from Comcast's peering link with Tata, Comcast would have to pay Tata (their transit provider) for the traffic.

Yes, the cost of the last-mile network is real and important, however, in this discussion it doesn't really relate, since you would be traversing the DOCSIS network in all cases.

[bjornsing]

Well... Isn't there a fundamental assumption in the network neutrality debate that cost per bit is zero or very low? At least that's the feeling I get from reading this discussion. People seem to think that cost comes mostly from IP transit, and implicitly that last-mile infrastructure cost is constant and sunk.

Well, that assumption is wrong for mobile, and to some extent for DOCSIS. Last-mile cost scales with traffic volume (and completely overshadows the cost of IP transit). That's why I think you'll see MNOs and MSOs fighting the hardest against network neutrality.

[wmf]

Level 3 is paying Comcast because the alternative was for Comcast to let the peering links congest, which would hurt Netflix streaming performance. Basically Comcast has a new business model for their Internet service: Everyone pays Comcast and Comcast pays no one.

[zbisch]

Yeah I can't believe this works for them. So I assume Comcast pays Tata, right? I was under the impression at some point that was their primary provider (paid transit). They keep these links so congested [1], that in order to actually provide reasonable service to Comcast users, a content provider needs to pay to peer with them (or pay someone else to) in order to give reasonable performance. I guess if you have enough users and enough of a demand to reach them faster, you can charge others. I take this to mean that Comcast should be paying me to subscribe to their Internet service! (or at least to watch Netflix on their network)

[1] http://www.merit.edu/mail.archives/nanog/msg15911.html

Edit: Although it is not my exact area of research, a colleague of mine has done some interesting work that discusses the costs of P2P traffic for some of these ISPs that is somewhat related: http://torrentfreak.com/large-isps-profit-fom-bittorrent-tra...