[throw14082020]

This is really helpful, thanks!

OpenAI hired the ex fractional CTO of LiveKit, who created Pion, a popular WebRTC library/tool.

I'd expect OpenAI to migrate off of LiveKit within 6 months. LiveKit is too expensive. Also, WebRTC is hard, and OpenAI now being a less open company will want to keep improvements to itself.

Not affiliated with any competitors, but I did work at a PaaS company similar to LiveKit but used Websockets instead.

[fidotron]

> LiveKit is too expensive

Most of it is open source, especially the clients, although they do feel quite ad hoc hacked together (a possible side effect of WebRTC evolution).

Would totally agree on OpenAI moving away. The description of the agent here sounds like a big hack just to get around the fact temporarily the model server expects audio over sockets instead.

[russ]

Which components feel ad hoc?

In most real applications, the agent has additional logic (function calling, RAG, etc) than simply relaying a stream to the model server. In those cases, you want it to be a separate service/component that can be independently scaled.

[fidotron]

Essentially I think the Livekit value is a SFU that works, with signalling, and the SDKs exist. My experience is people radically overstate how hard signalling is, and underestimate SFU complexity, especially with fast failover.

In terms of being a higher level API arguably it is doomed to failure, thanks to the madness of the domain. (The part that sticks in my mind is audio device switching on Android.) WebRTC products seem to always end up with the consumer needing to know way more of the internals than is healthy. As such I think once you are sufficiently good at using LiveKit you are less likely to pick it for your next product because you will be able to roll your own far more easily. That is unless the value you were getting from it actually was the SFU infrastructure and not the SDKs.

The OpenAI case is so point-to-point that doing WebRTC for that is, honestly, really not hard at all.

[russ]

You really don’t need to know about WebRTC at all when you use LiveKit. That’s largely thanks to the SDKs abstracting away all the complexity. Having good SDKs that work across every platform with consistent APIs is more valuable than the SFU imo. There are other options for SFUs and folks like Signal have rolled their own. Try to get WebRTC running on Apple Vision Pro or tvOS and let me know if that’s no big deal.

[fidotron]

> Try to get WebRTC running on Apple Vision Pro or tvOS and let me know if that’s no big deal.

[EDIT: I probably shouldn't mention that]. I have some experience of getting webrtc up on new platforms, and it's not as bad as all that. libwebrtc is a remarkably solid library, especially given the domain it's in.

I obviously do not share your opinion of the SDKs.

[russ]

Heh, actually I'm pretty sure I've come across your X profile before. :) You're definitely in a small minority of folks with a deep(er) understanding of WebRTC.

[russ]

Field CTO — hi @Sean-Der :wave:

Fractional CTO sounds like a disaster lol

[throw14082020]

My bad, he was Field CTO.