Hacker News new | ask | show | jobs
by pnash 3105 days ago
You've covered the "last mile", but there is the upstream world to take into consideration.

1) Connecting into a CO is one thing, but your fiber is going to need to connect into something. Who is paying for your optics, is there a port or line card that can accept those optics? Does the CO actually have enough bandwidth upstream? This is a real issue.

Back in the 90s, I helped set up an ISP in Boston proper and our main competition had well over a 1000 customers attached to a single T1 (1.5mb/s) link. Everyone wanted 28.8k speeds (lol), but would normally get ~300 to 2400bps. The competition had a bunch of modems with a single upstream link. No one wanted what we were selling - guaranteed bandwidth / true 28.8 bandwidth all the time. People wanted $19.99/unlimited all you can eat. People still want that today.

Back to the CO, maybe you are lucky and they have some open ports. Worst case scenario, they want you to plop down a router and you'll do 10gb/e between. You can go with a homemade box and hope that it is stable, or you can buy expensive network gear.

2) To your customers, that "CO" is the "internet", but to that vendor/telco, it is just a single point of presence (POP). That CO has to connect to other POPs that are owned by them, and that costs real money. Eventually through a unknown number of hops, your traffic will hit an exchange point or carrier hotel. This is where your traffic exits their network and is taken up by another provider and/or company (google has their own fiber plan, for example). The amount of bandwidth at these peering/access points is finite and providers choose to peer with each other, usually at no charge, if there is an equitable distribution of traffic. The last thing that you want is for one company to take up all of the (finite) bandwidth at a peering point.

An ISP connects to multiple carriers (l3, cogent, comcast, att,verizon, etc) so that your customers have quick access to the websites/services that they want to visit - which most likely have to traverse one of those other providers. Similarly, their customers will want to access services that you are hosting, so you will take in a similar amount of ingress traffic.

With the fiber network that you are connecting your customers to, they'll most likely want to access bandwidth intensive services. You better hope that your CO has upstream capacity and a fast path to netflix/hulu/facebook/google/akamai/etc.

Or you, as a internet service provider, try and peer directly with the content providers if they allow it. If there are only 2-3 hops between you and Netflix, your users will love you. If they have to bounce around the country a couple of times, your customers will go back to Comcast (because they have a well connected backbone).

3) This doesn't even cover where you are going to get your IP addresses, if your upstream provider will announce them in BGP for you, etc. Or maybe you connect in to two carriers, get an ASN and announce your networks yourself. You are still at the mercy of your upstream providers.

I think a lot of these details are often overlooked when someone talks about network neutrality. I think network neutrality is a glib term for a number of issues:

- filtering of traffic and/or inability to access a service - loss of freedom to host stuff "for free" on the internet - lack of competition in "the last mile".

The FCC/TitleII stuff, from what I've heard, negatively impacted small WISPs that were trying to start up, by assuming that they were the same size as major wireless providers. A $20k fine because your lawyer failed to properly submit paperwork can wipe you out if you are a simple provider that is trying to provide access to a small community. You aren't AT&T, but title II will assume that you are - and penalize you accordingly.

For more information, read some of these filings/papers:

https://ecfsapi.fcc.gov/file/10717113433056/FCC-17-108%20Res...

and in particular:

http://www.interisle.net/sub/FCC-14-28%20NN%20Interisle%20Co...

-Paul

2 comments

This is not correct.

Say you have a HOA with 100 houses and you got the last mile wired with fiber. There is probably some place ( such as community center ) that is owned by HOA itself. You get 100 pairs to that building. 10G LR SFP+ are $40 a pop all day. So you need $80 per link once. 48x 10G port switches are $3k all day. So it is 24x edges with a reasonable fabric oversubscription - so you need 5 of those because you want to oversubscribe core rather than the edge as edge requires interaction with a customer while core requires simple internal upgrades. In reality we are goig to do 1Gbit/sec to every drop delivered over 10G so we only need 100Gbit/sec to the edge. Lets spend another $10K on the "core switches" - which in reality are going to be the same as the edges but we will provision them in a way where should this take off we could replace core with 40 and 100G. All of this is going to cost us very little money. Hell, lets pretend it costs us $50K just for the sake of the argument because we like buying really expensive stuff

We can ride a single fiber pair ( remember, this is a residential service, so screw redundancy ) to one of the major interconnect centers because we can drop DWDM gear on our side ( prisms are cheap as hell ) and rent a rack in that interconnect location.

Monthlies:

$10K/mo ( worst case scenario ) DF to interconnect point $2.5K/mo ( rack at the interconnect point )

This gives us the L2 access. But that's not a problem. The problem is that 100Gbit/sec of non-congested IP transit is abou 55c per mbit/sec so that is $55K/mo.

So your cost is $67K/mo to provide 100 houses in a HOA with 1Gbit/sec of IP.

Lets say that you are in a magic place called say... NYC and it just happened that this wonderful thing is a building located right next to one of the big interconnect points and the developer who developed this highrise owns both buildings. You nuke dark fiber monthly cost. Hell, lets even pretend that the developer who owns both buildings lives in a building that we are wiring and he wants high speed internet connectivity to be able to watch NetFlix and PornTube. So there's not only no cost for dark fiber but there's no rack cost.

You are still at $55K/mo of non-congested IP to provide 1Gbit/sec access to every one of those 100 apartments.

Kind of insane to not oversubscribe residential or small/medium business connections, it's extremely rare that 100 houses would saturate a 10Gb line or even half that.

When you pay $50/mo for an internet connection you aren't paying for guaranteed bandwidth, you're just hoping the ISP has enough capacity to meet peak demand - not much different from your local electric provider.

It'd cost me roughly ~$3000/mo for a 10Gb point-to-point link from Boise to Equinix in Seattle from Zayo, and about another $2500/mo for a 10Gb transit connections from Hurricane Electric. You could serve quite a lot of households from that, 50-100:1 oversubscription is pretty common for residential/small business service - so that 10Gb connection could pretty safely serve 500 households reducing your fixed costs to $11/customer/mo.

Replying again.

What we need a boatload of small regional networks ( like the one with 100 houses of HOA ) that have an open peering policy. If you can peer out 50% of your traffic at $0.01 per mbit ($100/mo PNIs to CloudFlare, JoeSchmoeNet, FLIX etc) then you have the same non-congested non-oversubscribed exit for 50% less.

And this is where you are getting into some really interesting stuff:

what you want to do is be an ISP and content originator. In that case you effectively are double-selling your bandwidth since eyeball networks are bringing content in while web farms are pushing content out.

Oversubscription is a reality but it transparently works only on a very large scale - which is why Verizon and Comcast should be able to provide extremely high speed connections ( they don't due to their peering and interconnect policies but that's a separate thing ).

HE is terribly oversubscribed.

> Who is paying for your optics, is there a port or line card that can accept those optics? Does the CO actually have enough bandwidth upstream? This is a real issue. (snip) You better hope that your CO has upstream capacity and a fast path to netflix/hulu/facebook/google/akamai/etc.

Yes and yes. I don't want to dismiss this, it's a real need, but this is what you pay your upstream for. I've never seen a provider not take care of it.

I suppose if you cheap out on your upstream, this can be an issue. I can't imagine someone doing all the work to build a Fiber ISP, and then cheap out on the actual internet service, but I suppose anything is possible.

> This doesn't even cover where you are going to get your IP addresses, if your upstream provider will announce them in BGP for you, etc. Or maybe you connect in to two carriers, get an ASN and announce your networks yourself. You are still at the mercy of your upstream providers.

I don't know how common this is, but my upstream providers would just sell me the IP addresses and were flexible enough to handle either scenario.

> You are still at the mercy of your upstream providers.

Absolutely. This is always true, until you get large enough to be the upstream provider yourself and peer with others directly. But since upstream is competitive, and carries heavy contracts with teeth, you are mostly shielded from the worst atrocities.

It's kind of like forming a union. Sure, you're still "at the mercy of the employer", but you have way better bargaining power to prevent major problems, when you represent 10,000 internet users instead of just one. It's not perfect by any means. But it's worlds better than anything folks are used to on the residential side.

> The FCC/TitleII stuff, from what I've heard, negatively impacted small WISPs that were trying to start up,

Yes, fines should be lower for small business. But these guys could also just not break the law.

The complaints I've seen from some small WISPs are from people who are cheap and lazy, and want to do some pretty sketchy things. (Intentionally throttle Netflix to save upstream bandwidth, for example, because they want to sell 20mbps but can only provide 2mbps). These are blatant violations of Net Neutrality that would cause a shitstorm when AT&T/Comcast does it. But because they are 'small businesses', they want a bunch of sympathy despite doing the same slimy stuff.

I'm guessing there's probably an honest reason for some of the complaints, but the ones I've heard myself were all pretty shady. These providers give honest ISPs a bad name, and play into the false "everyone's just as evil as Comcast anyway" narrative.