[regularemployee]

I tried this myself too and when I try p2p with 4 people, out of 10 tests about 50% of the time I won't be able to see all 4 people or someone wouldn't be able to see all 4 people.

It was really hard to make p2p work and debugging the ice connections was even harder.

[Sean-Der]

WebRTC + networking is frustrating. IMO it is a leaky abstraction. There was a hope that ICE+TURN would work everywhere and users would never need to worry. That isn't true so we need to do a better job educating developers about what/why things went wrong.

I am working on a Open Source book that includes a WebRTC networking chapter[0]. Would love your opinions/feedback if this would have actually been helpful when learning this stuff!

[0] https://webrtcforthecurious.com/docs/03-connecting/

[parhamn]

Wonderful and very well written, thanks!

I too experimented with a p2p golang webchat setup. All the jargon was confusing and very hard to look up. This post has already given me much more clarity!!

[VWWHFSfQ]

iOS and Safari is riddled with WebRTC bugs like this. Sounds similar to my experience. Everything consistently works great in Chrome and Firefox and then only kinda works Safari. worst browser on the planet

[nobleach]

I've jokingly referred to Safari as SafarIE for the last 5 years. It does tick all these boxes:

1. Backed by an OS manufacturer that doesn't care about the web 2. Spends more time working on features that suit itself than meeting standards agreed upon by a body of which they're a part. 3. The only sanctioned/allowed browser on their platform (MS didn't even achieve this holy grail) 4. Lagging behind most other popular browsers by years in some cases

But due to it being the ONLY browser that'll run on iOS, I have no choice but to dumb down user experience for it. This year's lovely issue has been MediaRecorder - but supposedly that's made it into the most recent release.

[cjohansson]

The cycle repeats itself, maybe it's a successful pattern, that's why it keeps coming back..

[NetOpWibby]

SafarIE?!

Bahaha, brilliant.

[mguerville]

I use firefox on iOS, it’s even set as my default browser. Curious what you mean by “safari is the only browser that works in iOS”? For in browser video chat? Genuinely curious, what are the limitations?

[danielheath]

Firefox on iOS uses the safari js runtime and the safari layout engine.

[mguerville]

Ah I see, thanks I had no idea (I assume most average users are in the same situation). I assume IpadOS is the same, and only MacOS allows a true non safari browser.

[VWWHFSfQ]

Firefox and chrome on iOS are just re-skins of safari. Apple doesn't allow other browser backends

[danielheath]

To elaborate on "doesn't allow":

iOS binaries that are not signed by apple are not permitted to mark memory pages as executable if they have previously been writable.

This restriction makes exploiting buffer overruns very difficult on iOS - particularly important as objective-c doesn't give you much help avoiding them.

However, you can't write a runtime compiler unless you can generate bytecode (write) and then execute it, and nobody has found a way to write a performant javascript or CSS engine without some form of runtime compilation.

So, Apple does allow you to write your own browser backends, but they won't give their signature, which would permit you to use riskier techniques to gain performance.

In practice, that means any browser not using the safari engines would be unacceptably slow on the modern web.

[nobleach]

This is the absolute best (and correct) explanation I've ever seen in such a small space. Thank you.

[yawnxyz]

I thought I was just dumb for not being able to figure this out, but turns out it's just a hard problem for everyone haha

[6510]

Yeah, it made me feel like I had to read the manual but there wasn't any. Seeing such a bad one reminded me how spoiled we are with beautiful abstractions.

[deneb150]

Debugging this is indeed quite hard. I have only used it for 1 on 1 p2p calls.

[moron4hire]

That's probably Network Address Translation (NAT), which requires TURN (a fancy name for a central relay for all media) to "punch through". TURN literally stands for "Traversal Using Relay around NAT". And it's just a traditional, centralized. non-p2p fallback for people on paternalistic networks that don't allow them to create UDP connections or TCP connections on any ports other than 80 or 443.

Which, as it turns out, is a lot of users. I've seen estimates in the range of 10 to 20% of users. Which means, for a random selection of 7 users, you pretty much have a 50/50 chance of not being able to peer everyone using just STUN.

[kwindla]

I think it is likely also bandwidth and cpu issues with mesh peer-to-peer.

Unless you're capping the video bitrate, the browser will try to use whatever the browser's default target is, for each connection. On Chrome that's 3mb/s, which is a lot of network bandwidth, and turns out to be a lot of cpu as well just shuffling those packets through the encoding->sending->bandwidth-estimation and receiving->decoding->rendering pipelines.

Capping the video bitrate is more complicated and confusing than it should be. It's better now that the browser implementations are all more or less closing in on "WebRTC 1.0" compliance. But you still need to reach into either the raw SDP you are exchanging during signaling, or the RTCPeerConnection objects, and set the encoding bitrate target.

The SaaS platforms that offer WebRTC APIs and infrastructure all do a lot of work under the covers to set bitrate caps, track constraints (resolution, for example), and other bits and pieces of WebRTC config that work well on a wide variety of networks, devices, and browsers.

[Sean-Der]

The only numbers I have ever seen published are Whereby's[0] they saw 17% used TURN.

There is a little more nuance then just paternalistic networks though. In same cases like NAT Mapping exhaustion you just can't give an individual user multiple long lived mappings. Address Dependendent filtering/mapping also makes sense in some cases. It makes P2P harder, but does give you the ability to provide your users more sessions at least!

https://medium.com/the-making-of-whereby/what-kind-of-turn-s...

[kwindla]

We see about the same numbers Whereby does at Daily, globally across our whole user base. Bounces around a little but is usually just under 20%.

Way more for customers that are mostly serving corporate users, of course (firewalls). And more for mobile-heavy user populations.

Actually, that's a good reminder that it would be nice to understand the mobile data networks breakdown in more detail. Most of the US mobile data networks require TURN, as far as I remember when I last looked at this. But I don't know if that's true everywhere in the world.

[Sean-Der]

That is awesome, thanks for sharing :) Lots of little details and they all effect each other. I really enjoy networking because of this.

I wonder if we come back to this in 10 years what this number will be. Linux on the desktop and IPv6 is just around the corner...

[mekkkkkk]

Thinking about it, v4 addresses and oil have a lot in common. The exhaustion/depletion is coming, but we keep finding ways to circumvent it. For oil you got fracking and sand. For v4 you see wider adoption of wide/carrier-grade NAT. In both cases it temporarily solves the supply problem really effectively. However it also ruins the environment.

[throw14082020]

I always assumed everyone is behind NAT, you're saying on 10 to 20% of people are, and therefore only they need TURN. I'd love to see where you got that number.

If I were to guess, the problem GP is facing is bandwidth, a mesh network uses exponentially more bandwidth. For each user, the bandwidth is linear, N more people requires N more bandwidth. This is fine for downloads, but uploading N more can be much more challenging for certain networks.

[cma]

He's mistaken that NAT always requires TURN. Consumer NAT typically still allows incoming UDP, using STUN/punch-through, or TCP with uPNP support. He maybe meant to talk about only about more restrictive NAT situations or campus/corporate/ISP/nation-state/scientology-compound firewalls.

[throw14082020]

Im not sure about UDP hole punching and how it relates to WebRTC, i don't see it being talked about much. In general hole punching is a rare thing to hear, I cannot find much resources about it.

And uPNP (Universal Plug and Play) sounds like its for device discovery in the same local network, so again, it doesn't sound related to webRTC, we can connect directly with each other on the same local network anyway.

[mypalmike]

Hole punching in the context of webrtc is usually referred to as ICE.

UPNP has a number of functions, including forwarding of WAN packets to a specific LAN device.

[mypalmike]

The 10-20 percent would reflect people with symmetric NAT, which is rather common with mobile networks, corporate NAT, etc. Symmetric NAT requires TURN relays. Typical home router configurations are not symmetric NAT, and usually work with just STUN.

[meheleventyone]

The worst one for me so far is mDNS not working on my local network so the one circumstance you should basically be able to guarantee an easy P2P connection doesn’t work.