[tc]

Since I'm in this industry, I'd like to point out that bandwidth is not artificially scarce. It is just scarce, necessarily limited by the capital available to buy equipment and lease facilities, fiber, and copper. Before talking about the artificial scarcity of bandwidth, I'd encourage you to check the prices on some Cisco or Juniper gear. No one buys this stuff to intentionally underutilized its capacity. And correspondingly, no one avoids buying this gear to create scarcity; they avoid buying it because it is expensive.

Carriers are bound by the same forces as other businesses. People buy expensive equipment if and only if they expect to see a return on their investment.

[skolor]

I am fairly aware what network infrastructure costs. I am also very aware that in several other countries it is not only profitable, but profitable enough that there are several companies doing it.

There are only two reasons I can see for not having the kind of high speed connections some many other places in the world enjoy, is either: A) the backbone of the network is overloaded, and that is what needs upgrading most, or B) they don't want to upgrade one part of the network and leaving the non-urban areas with only lower speed connections. Without any kind of inside knowledge, I would guess it is a mixture of the two.

In any case: I pay roughly the same as I could get several times the bandwidth in other countries. If it is economically feasible there, there really is no reason why it wouldn't be feasible here. You may not be able to see network-wide improvements due to a slow infrastructure, but with sites like Google and Amazon putting up edge servers in most large cities, and the use of P2P technology, the gains would still be significant. Hell, if the major ISPs really wanted to make an impact as far as bandwidth was concerned, they would embrace P2P far more. A network-aware P2P network helps solve much of the problem of a slow backbone connection, as long as there is a fast local (metro-area) connection. Of course, I can't see the companies going for that, for the same reason they dislike Net Neutrality: it forces them to give up control.

[tc]

"Without any kind of inside knowledge, I would guess"

I don't want to be overly-harsh to you in particular, but I think this is the key to so much of the useless posturing and speculating on this issue.

It may in fact be similar to the familiar refrain that people become more tolerant of homosexuals when they realize that one of their friends, family members, or business partners is gay. It's easy to stigmatize and demonize internet carriers if you don't know anyone who runs a business in that field.

To your points though. Most carrier backbones are not regularly overloaded. The bottleneck is at the network edge. And it isn't that carriers don't want to build. Everyone I know in this industry loves expanding their network and increasing capacity. But at the end of the day, you have to make a return on investment or your go out of business. Building out the network edge is terribly expensive. You have to buy and place a lot of expensive equipment in a lot of different places. You have to lay or lease lots of fiber or copper, or you have to build towers (and lay fiber to backhaul). Amazingly, there are lots of carriers that do this. You wouldn't even recognize most of the names.

Speaking as someone who came to telecom from a software background, I think there is a tendency for software people to want to idealize telecom into an ethereal mass-less abstraction, like software. If you look at it that way, any limitation is artificial -- the man is holding you down for his own pleasure. But the business isn't anything like that. Think Fedex rather than Google, and you'd be on the right path.

As for Europe, Asia, etc., keep in mind that those networks were far more heavily subsidized than our own. You may think that's a good thing, but remember that this removed resources from other human endeavors. Not everyone on Earth rates decreasing his or her CounterStrike latency as the highest priority in life.

[skolor]

If you don't mind educating me a little, what you have described is the exact opposite from what I have heard, both from people inside the telecom industry and outside of it. Feel free to correct me at any point, but this is how I understand the current state of things:

There are two observable forms of "internet speed": latency and throughput. Latency is the time it takes for a packet to go from your computer to the server and back, and is (usually) measured in milliseconds. The other type is throughput, the speeds that are usually advertised by ISPs, measured in mega(bits|bytes) per second.

From my understanding, there is little that can be done with latency. You can minimize it, but it is prohibitively expensive, for not much of an increase, especially when you consider that a good portion of the observable latency comes from the server processing the request. Instead, ISPs (and end users) focus on throughput as the primary measurement of speed.

Throughput is relatively easy to increase. The bigger the pipe, the more requests per second your computer can make, and the more data can come down the pipe at a time. Every few years some new advances are made, and you can lay down a new pipe (albeit at rather high cost) and double your throughput.

So, with the two types of speed, there are two types of traffic. One is loading webpages, and regular internet surfing. You may a request for a relatively small set of files (the website you are trying to view, usually), and the magic fairies in the tubes find it and bring it back to you. On the other hand, you have throughput intensive traffic. Things like VOIP, Streaming music, or any kind of large downloads. P2P falls into this category, but its a little different. With largely transfers like that, the latency becomes insignificant. After the transfer begins, it is going to go more or less un-interrupted until it finishes. The only real effect that the network speed has on it is the throughput, determining how much of that data can come down the pipe at a time.

Now, from my understanding (as someone not in the field), the problems arise from traffic trying to go over the backbone. While the routers and such supplying to backbone are more than sufficient to keep up the latency, the limitations come when you get a whole lot of people streaming videos (for instance) at the same time. The backbone, while it can switch packets quickly, has (again, from my understanding) a relatively slow throughput. While it is more than sufficient most of the time, the problems arise when everyone wants to see that new hamster dancing video on Youtube, the backbone just does not have that kind of throughput to serve much more than a fraction of the users at the full capacity of their connection at a time.

That is, how I understand it, the root of the problem. That would be simple to fix, if anything could possibly be simple to fix when it comes to networks. The problem arises from the fac that there isn't one big mystical Backbone owned by AT&T or Comcast, that everyone else connects to. Instead, it is made up of companies like Level3 and Cogent, which have peering agreements with each other. Since they're both companies, they both have bottom lines to meet, and they're primarily out to make a profit, not to make a nice internet for the users.

That brings me to the other point. I was under the impression that other places in the world had received far FEWER subsidizations of their network, and that prices were kept down, and speeds up, because there was far more competition in the markets. Admittedly, that is largely because the connect areas in Asia and Europe are considerably smaller, and cheaper for the first investments. Becase it was cheaper to get started, more companies did, so they got considerably larger and more complex peering networks, Each network had to have an edge, so they were either faster, or cheaper.

While the network companies want to expand their networks to more areas, and serve more people, what the users want is a faster connection. This leads to the whole problem with net neutrality. The only way for the consumer ISPs to get their networks faster for the users is to either lay lots of more fiber of their own, spend a lot more on faster peering agreements, or just try and stop the users who use the most bandwidth to cut it out. As far as I can tell, the primary argument against Net Neutrality is something along the lines of: The peering centers do it to us, why can't we do it to the users?

So, to summarize: my understanding is that the big problems arise from peering agreements. It is in no ones best interest except for the end user to heavily increase the backbone connections. It has no advertising potential like increasing speed at the edges does, and it has little profit potential, since most of the large peering companies have a near monopoly for their regions/portions of the internet. They can simply say: Pay us for our slow speeds, or go lay your own wires. There is no real reason to upgrade the network, because no one else has. You can't get an increase if only one segment is faster.

Did I get at least fairly close to it?

[niels_olson]

> those networks were far more heavily subsidized than our own.

BTW, didn't the US give the telecoms several billion to build infrastructure? Where did that go? And how far did it go? How much does it cost?

Let's say some bankers and lawyers and doctors buy out a 10,000 acre farm in coastal Virginia, 5 miles from the nearest stoplight of a medium-sized city, and they want gigabit ethernet to their McMansions and 3G coverage all the way into work. Essentially, a lollipop-shaped extension of network hardware into previously unpopulated land. What's it going to cost?

[tc]

Look, I don't support subsidies either. Like with most US subsidies, the money was probably mostly wasted (but I'm sure it created jobs).

In all seriousness, telecom subsidies in the US mostly go to building out old-fashioned POTS lines and slow DSL into corn fields in the middle of nowhere. Don't blame the telecoms (except for the lobbying of the huge ones), that's what the grants specify.

AT&T has been spending big money supporting and astroturfing Internet Neutrality. If you don't understand why that would be, you don't understand the dynamics of the industry enough to know what is really going on. They aren't doing that out of the goodness of their hearts. This is the old struggle between the LECs and the cable franchises.

Internet Neutrality is a win for AT&T and the LECs because they already have a separate network (the time-division network) that doesn't rely on packet prioritization to deliver latency-sensitive media.

[tptacek]

High tens of millions of dollars is my guess, between right-of-ways and spectrum licensing. The routers themselves might not even break a million, though.

[tc]

Yes. Building out fiber to a single point would be the easy part (as low as hundreds of thousands of dollars, perhaps). The continuous 3G coverage would be expensive though.

[derefr]

So, then, what makes building out last-mile capacity in NY or SF so much harder/less ROIful than building it in Tokyo?

[tc]

Remember Japan's "lost decade"? The government threw money at every infrastructure project they could find.

Before you get too nostalgic about Japanese internet though, a friend who worked in Tokyo for a couple of months tells me it isn't really that much to write home about. Access to all of the services you care about as an American are slower, and my friend's provider explicitly blocked VoIP. I have no doubt that this isn't fully representative (except the latency/limited intercontinental bandwidth part). Perhaps patio11 could lend some experiences?

[skolor]

But we aren't just talking about upgrades that were made in 90's. The network there is currently significantly faster than it is here. Even if it was primarily jump-started by large government funding, something kept it going, and it has kept growing while our infrastructure (at least as far as an end user is seeing) hasn't seen much of an upgrade. From what I can tell, US citizens have seen an upgrade of 10-30x what speeds were at the end of the 90's. Places like Tokyo have seen increases of 40-100x since then.

[niels_olson]

No knock on you in particular, but the comment is typical of a sort of insider Stockholm syndrome where members of one industry, or one sector of an industry are presently at the mercy of another sector, and the "prisoner" tells outsider "Well, from the outside, idea B might appear attractive, but if you have my inside information, you'll see Plan A is Plan A for good reason."

Your comment does not address why the equipment is expensive (Cisco exerting oligarichic price controls?), or why the market won't bear added expense (eg, where do so many other countries find the money?)

[tptacek]

Cisco gear is expensive because it is expensive to design and expensive to manufacture equipment that can shuttle frames between multiple OC-192 circuits at sub-millisecond speeds across a purpose-built backplane that won't lock up under the load, and because it is even more expensive to design the software that will manage the forwarding databases on that equipment under the strain of multiple global routing table updates every second without ever hiccupping, and because for all that effort, you can count the number of US companies that will buy that kind of a product in any real volume on two hands.

It's a weird argument about the scarcity of bandwidth that suggests that it's all a price fixing scam by Cisco. Cisco, for what it's worth, does not make the majority of its money on big-ass backbone routers.

It is, in fact, rather akin to the idea that Southwest Airlines tickets cost $200 because Boeing has artificially inflated the price of the 737.

[niels_olson]

> It's a weird argument about the scarcity of bandwidth that suggests that it's all a price fixing scam by Cisco

I heartily agree, but that is the thesis tc postulated in his comment, to which I responded.

[tc]

I think using the Stockholm syndrome is an absurd metaphor in this case. In the Stockholm syndrome, the victim defends the attacker. In this case, the 'attacker' is the regulation, and I'm opposing it.

Seriously though, I think I've addressed some of those questions elsewhere in the thread. Many other countries you would compare against have heavily subsidized consumer internet via government spending. Many countries also have a significantly higher population density than the US, or they had less existing infrastructure (laying fiber is cheaper if you are building all new roads anyway).

As for why Cisco or Juniper equipment is expensive: it is expensive because it is relatively low-volume industrial-strength gear that needs to run far in excess of 99.999% reliability while supporting an absurdly large number of protocols and standards. To put the problem as a bit of a tautology, if you think it is so absurdly overpriced, why aren't you raising a venture round to build cheaper gear?

(Incidentally, I think you probably could, but it is still a hard problem that requires time to solve. Not all things that could theoretically be improved can be improved instantly.)

[niels_olson]

> I think using the Stockholm syndrome is an absurd metaphor in this case. In the Stockholm syndrome, the victim defends the attacker.

fair enough. It was a convoluted enough metaphor that I've since forgotten how I stitched it together. Anyway, I appreciate all your information in this thread.

[tptacek]

I mean, hey, it worked for Procket!