Lack of a central server does not imply end-to-end security. In just about any useful situating, when A connects to B, the following will at least be in between: A's modem, A's ISP (typically multiple routing devices), B's ISP (ditto), B's modem. Since all of these are literally in between, they could act as men-in-the-middle... barring countermeasures.
End-to-end implies some crypto scheme whereby all intermediate points can only act all-or-nothing: either they forward packages or not. The contents of the packages should be inscrutable to them. (ideally, you'd want the meta data to be similarly indecipherable, but that requires a Tor-like infrastructure.)
So, either there is some cryptography, which requires exchanging keys somehow, or it's not end-to-end in a meaningful way.
WebRTC is end-to-end encrypted by default. There is a signaling server that helps establishing the connections between users in a room, but after that the communication is encrypted. Also those TURN and STUN servers are only required for technical reasons to get peer-to-peer working. So no content is ever passed unencrypted.
That's the difference to other services like Zoom and Jitsi, where a server in the middle is receiving the video streams unencrypted and then redistributes. Although Jitsi is adding encryption support for that as well soon.
I think you are very much misinformed on the subject.
You're not sending unencrypted streams to Jitsi, as it uses hop-by-hop encryption, meaning you send an encrypted stream to the server where it gets decrypted and re-encrypted for the rest of the participants. At no point you are sending "plaintext" data. What you might have heard is that Jitsi will be introducing end-to-end encryption, but it doesn't mean it doesn't have encryption at the moment.[1]
I think it's a misnomer to say WebRTC end to end encrypted by default.
If you cannot guarantee end to end encryption during the negotiation phase, you cannot trust that the party you're encrypting for/receiving encrypted content from is the actual party you are expecting to communicate with, hence the need to independently verify over a trusted channel whether the fingerprint in the SDP offer/answer matches.
After the negotiation is complete, the streams will indeed be encrypted, but if the offer/answer SDP has been tampered with by the signaling server, which is under the control of a rogue/compelled provider, and another DTLS certificate has been inserted, the encryption is useless as far as complete communication security is concerned.
I did not want to say Jitsi is bad. It is one of the best solutions around. But you still have to trust the server if e2ee is not turned on and more than 2 participants are using it [1]: "This outer layer of DTLS-SRTP encryption is removed while packets are traversing Jitsi Videobridge"
I believe it is correct to say, that WebRTC is end-to-end encrypted. What you are referring to is, if you can trust to have a direct connection to the other participant and that you can make sure she is the person she pretends to be. This layer is indeed missing and will be added [2]
I would very much appreciate your help on the project to eliminate those concerns.
> This outer layer of DTLS-SRTP encryption is removed while packets are traversing Jitsi Videobridge
What this means is that the packets are encrypted between you and the video bridge and the packets travel unencrypted within the videobridge infrastructure, but at no point does the outside world see the unencrypted stream. Talking to the Jitsi video bridge is just a regular WebRTC session as far as clients are concerned (so any DTLS encryption WebRTC has, the Jitsi call has), I know this because before they had a decent SDK I had to manually wedge my WebRTC calls into it.
If you're looking to add e2ee to your project, you're pretty much left to choose between Olm and Signal Protocol, which are quite similar, but libsignal is GPLv3 while libolm is Apache 2.0.
I've used Olm more intensively over the last two years. While it works, it's great, when it randomly wedges a session for no apparent reason you feel like throwing it out the window.
End-to-end implies some crypto scheme whereby all intermediate points can only act all-or-nothing: either they forward packages or not. The contents of the packages should be inscrutable to them. (ideally, you'd want the meta data to be similarly indecipherable, but that requires a Tor-like infrastructure.)
So, either there is some cryptography, which requires exchanging keys somehow, or it's not end-to-end in a meaningful way.