[nycmesh]

No you don't need hundreds of thousands. You do need tens of thousands. We've had no downtime this year at supernode1. Yes reliability is a big priority, but it is not a big priority for the big ISPs in NYC. Time Warner Cable (now Spectrum) has barely two nines. In Manhattan it would go down in the East Village after every large storm. We are already providing faster more reliable connections than this.

[mediocrejoker]

Thanks. This seems like a cool project.

Do you know what the cost per user is currently?

What frequency do the point-to-point links run at? Have you considered using some sort of lower frequency packet radio for longer distance links, perhaps to other nearby nodes?

[nycmesh]

Our recurring costs are only for the lease at supernode1, which is around $1000/month. Divide that by our member node count (158) multiplied by users of each node ~4, 1000/(158*4)gives you under $2/month per user. Members usually pay for their own routers so that isn't included. To set up the supernode was about a $10K one-off charge for install fees, servers and antennas.

DE-CIX, our IXP, donated bandwidth to us, and we have transit also donated from Packet Host and WebAir. We actually pay nothing for bandwidth. Probably one day we will pay but it is not that expensive if you do this at an IXP and use peering.

We have an AirFiber pair that is 24Ghz, but all of our sectors are 5Ghz wifi.

[eric_h]

> DE-CIX, our IXP, donated bandwidth to us, and we have transit also donated from Packet Host and WebAir.

What is the incentive for them to donate bandwidth/transit to you? Your non-profit status? Do they get something else in return?

[nycmesh]

We get a lot of support from the NYC networking community, and we have a lot of friends at the NYNOG meetups. Everyone is very helpful and they also want the big ISPs to have some competition, so we get donations from quite a few people. Some also join our network and get nodes on their own roofs and help with other installs!

DE-CIX is the biggest IXP and they measure their network in terabits/sec. It is not a big deal for them to donate us a 1 gig connection. (it's a big deal for us!) Eventually we may upgrade it to 10 gig.

[bigiain]

So you're sharing a 1G connection between 600-ish users - all presumably on 802.11ac 450Mbps (or 1.3Gbps if they've got three-stream?) connections?

How often do you see full saturation on that link? I'm guessing if all 600 of your users all tried to stream one stream of $TV-show-de-jour at once they'd barely get 16Mbps each? Can one user's home office full of Apple gear flood the bandwidth downloading a dozen Apple software updates simultaneously?

[StudentStuff]

That is a pretty low ratio in terms of over-subscription, on many cable networks you'd have 24 downstream Docsis channels @ 38Mbps usable per channel, for 912Mbps usable across all houses on said node. A single node often supports 500 homes, and that 912Mbps of capacity carries switched digital video, voice & data traffic.

Cable companies consider these high ratios to be quite embarrassing actually, as it shows how crummy their networks are: https://www.dslreports.com/forum/r31258251-Speed-Issues-Char...

[nycmesh]

Netflix standard streaming is 3Mbps, so even with your math (which isn't how bandwidth works) everyone is happy.

We've never come anywhere near saturating our gig connection. We monitor it. Each building is limited to their own rooftop connection which is an average 100Mbps, so they can't do more than that if they tried. The thing is not everyone is downloading a file at the same time, and streaming uses much less bandwidth than downloading. From Netflix site-

0.5 Megabits per second - Required broadband connection speed

1.5 Megabits per second - Recommended broadband connection speed

3.0 Megabits per second - Recommended for SD quality

5.0 Megabits per second - Recommended for HD quality

25 Megabits per second - Recommended for Ultra HD quality

[ybloviator]

Ah, so you get free bandwidth from people that want to balance their traffic for better peering contracts. That's clever. Probably not a helpful model for other areas that aren't bandwidth centers like NYC, and should probably be more out there when you're evangelizing so people understand you're getting a large portion of your operating expenses donated...

I have to assume another cost that's not noted is roof rights. That's not free - most people either have to pay or give the property management a kickback.

NYC is an interesting place for this - if you live in Manhattan, you clearly can afford the $80 for 1Gb/s FiOS or $40 for 100Mb/s FiOS or whatever Spectrum is charging. It would be way more interesting to plop this in a rust-belt city where people are on a paycheck-to-paycheck salary...