[heavenlyhash]

Lots of people -- in that list, even -- started with XMPP based systems. They all move away. At first, I also thought this was an open and shut case of interest in lock-in. Now, I'm not going to say that isn't true, but it's also worth considering that XMPP... just isn't that good.

Having recently started working on a chat client that was intended to speak first-and-foremost XMPP, I have to admit: XMPP is... it's decades out of date and missing a number of absolutely critical, central ideas that are essential to making stuff work. I can't honestly fault anyone who started with XMPP and then rapidly started looking to get off.

Examples:

- unique message IDs? Absent. XEPs kind of provide; but I can't tell you which of the three or for relevant ones are the most relevant (AMP IDs from XEP 0079? Stream Management from XEP 0198? Acking from XEP 0184? Something from Carbons or MAM in 0313 or 0280? You know, if you wanted some light reading...).

- multi device? Oh. My. God. It's bananas. The spec behavior is that whenever a client sends a message, the server is supposed to consider that one the most alive, and then route all future messages exclusively to that one. So you send a message on your phone? Yeah, your desktop is just going to silently stop receiving messages.

- there's a concept of "message carbons" to deal with this. This involves re-sending all your messages back to the server after you receive them, with special instructions to send them back again to your other clients. The amount of redundantly redundant XML involved is eyewatering.

Combine that multidevice behavior (messages can get randomly routed anywhere at any time) with the wild-west nature of message delivery acks, and you can see how ridiculously difficult this makes the basic idea of "all clients should see the same picture".

Overall, the XEP process, conceptually, is a great example of open extensibility. The trouble is, so much of this stuff is core to sane message delivery semantics that it really, practically speaking, causes huge problems when it's all considered "extensions". Stuff like message IDs fundamentally shouldn't be an extension because it's just too critical that all minimum-viable clients agree. You just can't build higher level stuff without that. XEPs are great. A community process for extensions should exist. It just needs to exist for extensions, not subsume the total set of realistic minimum viable features.

I want an open chat protocol.

I thought XMPP might be the one. Then I actually looked at it. Tried to write something with it.

XMPP is not prepared to be the one, in any sense at all except for legacy support. I have all the respect in the world for all the folks who put effort into trying to make an open chat standard, but... honestly, we need a stronger foundation than this.

(N.b. These complaints are a shameless repost from another not-very-old thread about chat where XMPP was also raised: https://news.ycombinator.com/item?id=9718784 .)

I've since looked at matrix.org ... and I'm really impressed. The standard is completely reasonable. It's the first federated chat standard I've seen proposed that actually has loose timing systems that's both available during a partition and can reach consistency afterward (messages list their last-seen messages, and so act like vector clocks, an accepted good solution to problems in CAP territory). You could actually take two systems with clock skew and have them reach a single, reconciled view of the world, even after netsplits. And practically speaking... man, go look at their list of clients. Despite being a relatively young project, matrix has clients on every major platform -- as far as I can tell, this is better platform coverage than XMPP, and better feature parity within those clients since critical things like message IDs are actually baked into the spec. I really suggest giving matrix a look. There's a lot going right over there.

[Flowdalic]

You are absolutely right. I usually encourage people to get involved in improving the (XMPP) situation.

Just last week I submitted a XEP for unique stanza IDs to the XSF [1], which will most likely be the base for IDs in MAM and other protocols. I guess that's the light reading you want. :)

And yes, Carbons (XEP-280), in it's current state, also suck. I'm working on a replacement XEP called "Message Routing 2.0 " (MR2): http://geekplace.eu/xeps/xep-mr2/xep-mr2.html It's in an early draft state, but I think it addresses most, if not all, issues with carbons, improving the multi-device situation in XMPP somewhat (I don't care if it's MR2 or an improved version of carbons that makes the race).

> see how ridiculously difficult this makes the basic idea of "all clients should see the same picture" Being involved in XMPP a bit, I get quite a few requests on how to implement WhatsApp/Hangouts like groupchats. And while the basic idea is quite simple, implementing it is a non-trivial task. But it can certainly be done. But we need more motivated developers and specification designers to get it done. While the XMPP community is just great, the count of activley involved people is quite small.

1: http://mail.jabber.org/pipermail/standards/2015-June/029865....

[lewisl9029]

One thing I'm curious about regarding matrix.org:

As a federated protocol, is there anything at all in the spec that would discourage someone from piggy-backing on the federation to acquire initial users, goodwill and a head-start on features, and then locking users in once they've gained enough momentum by making proprietary modifications?

You know, the same tactic countless companies have pulled with XMPP?

I've been eyeing matrix.org for quite a while now, but it seems to still lack a "killer" client app. One that can at least match the features, polish, stability, usability and platform ubiquity of existing proprietary messaging apps, an app that can help matrix.org grow user mindshare and keep it.

The open source community just doesn't seem to do a very good job of churning out these kinds of consumer-oriented apps. And if some company ends up building such a "killer" app for matrix.org and gains momentum, they'd have some very sweet incentives to lock in their userbase.

[Arathorn]

Oh, in terms of the risk of federation causing lock-in: it's inevitable that folks will add domain-specific extensions to try to force people to use their platform. This is no different to me sending you an attachment over email in some proprietary vendor-specific format (e.g. .ppt) which obliges you to install PowerPoint to read it.

This can be used for both good and evil. The evil scenario is above - rather than using an open standard like ODF, it provides a route to promote vendor lock-in. The good scenario is that it provides extensibility for technology that simply isn't standardised yet, and gives vendors a way to differentiate their product. For instance, if Oculus jumped on Matrix and started using it to negotiate VR collaboration spaces, it almost certainly wouldn't work with any other vendor at first... but it's good news for end-users who get a cool feature which some day may be standardised across all of Matrix.

The bottom line is that as long as everyone implements the common base line use cases in an interoperable fashion (i.e. IM and VoIP), then vendors having freedom to put proprietary/experimental stuff on top is just a necessary evil... as long as they don't break the baseline. It's up to us as consumers to then encourage vendors to make their extensions standards rather than exploit them for vendor-lockin. It really is an identical situation to email and MIME.

[Arathorn]

The example 'Matrix Console' iOS & Android apps are aesthetically very ugly, but still very functional and usable from a geek perspective. Meanwhile we're currently replacing the example Matrix Angular webapp with a Matrix React webapp which will improve the performance enormously.

In terms of polished UI/UX, we were hoping the community would fork the example apps and put the necessary polish and usability on them. Features should (almost) all be there, and we're doing our best on platform ubiquity.

However, as ex3ndr's comment below shows, there are people still creating their own per-app proprietary protocols rather than building on something like Matrix.

So in order to help bootstrap Matrix, am sure we'll end up building a killer client app sooner or later if nobody else does :)

[lewisl9029]

Thank you for the detailed reply.

Glad to hear you guys have plans to build a client app if needed.

Regarding matrix home servers, are users expected to just trust that they won't read/leak/share their private data (contacts, chat history, profile info, etc)? Or do home servers store user data in an encrypted form so users can just treat them as a zero-knowledge storage server without having to trust them at all? (I couldn't find storage encryption mentioned anywhere in the spec or anywhere on the faq, so I'm assuming it's the former)

If it is the former, are there any plans to move towards the latter? As we've seen in the case of email, even federated services tend to eventually gravitate towards a usage model where a handful of very large service providers service an overwhelming majority of users. I don't think allowing these service providers access to the private data of all of their users is in the best interest of the matrix community.

[ara4n]

For now, it's an implementation detail (rather than specced) as to how homeservers persist their state. The synapse implementation stores all data unencrypted in sqlite or postgres - /however/ that data may be end-to-end encrypted (we are releasing our e2e support over the course of this week - the first bit of the puzzle can be seen at http://github.com/matrix-org/olm). We should probably store all data AESed in the db to avoid casual snooping too.

This doesn't obfuscate metadata like room membership or profile data however; but fixing this is Hard. For now it's just a fact of life that Matrix servers have visibility on communication metadata - i.e. the identities of who talks to who, and when, and with what kind of event. In future we may support better privacy preserving semantics by evolving the federation architecture: eg running homeservers on clients and using Pond-style hidden Tor services for message transport, or layering on GNUnet as a transport. We've tried to design Matrix to support this sort of evolution, but right now today Matrix provides the same level of metadata privacy ss (say) an IMAP or SMTP server.

[ex3ndr]

Can't agree more. That's why we started our project with fresh and new protocol for messaging with implementing all required stuff in it's core, not extensions.

[Arathorn]

I'm not sure that creating a new vendor-specific protocol without federation or cross-vendor interoperability really helps anyone but you ;)

[bad_user]

Hey, thanks for your rant, very informative :-)

As a user, I don't care which standard gets implemented, I just want one now.

[heavenlyhash]

Agreed :) Though I've been playing with matrix's stuff, I'm not wearing anybody's jerseys; I just want a standard.

I have to share those impressions of XMPP because I'm afraid XMPP is very much "good enough being the enemy of good". I used to be in the camp of "what good is [new standard X], didn't they read about XMPP?" and now... well, now I see that there's a reason XMPP adoption is faltering, and it really does require going back to the drawing board. XMPP was great for the time, but at this point I also don't want it to take steam away from folks who are making a real shot at a better standard.

[mickael]

All the flaws you mentioned can be solved. We solved them in ejabberd.

[davecridland]

Matrix do have a much bigger marketing budget than the XSF, it's true. They fly their employees all over the globe, and appear to have a constant stream of FUD on tap about XMPP, presumably because it's their strongest competitor. Last I saw, though, it's got a thumping great single-point-of-control in the middle, firmly under their corporate grasp. Nothing too critical, though, only your identity. (Looking at the FAQ, the "Can I run my own Identity Server?" question is simply missing, but I can answer that one: No)

XMPP's primary distinction is that all services are fully partitioned by domain. The only shared state - and I use this term very loosely - is in subscription data, which is long term. (By subscription data, I'm intending to mean the roster, pubsub, and also MUC chatroom occupancy). Matrix attempts to share considerably more state, acting much more like a single chat service scattered across multiple servers, but due to the centralized and privileged identity servers, there's a significant loss of autonomy in the system. Important to note, an XMPPservice on a single domain can be spread across multiple physical servers, fully sharing state, in which case you're well into the territory of vector clocking and CAP theorums, but that's an internal matter for your service and doesn't affect the inter-domain federation at all.

In layman's terms, you can run an XMPP service without reference to any external system - although federation would need you to reference DNS. Running a Matrix server leaves you somewhat under the control of Matrix, which despite appearances is just a branding for a commercial outfit.

Well, that, and XMPP is an open standard according to a definition we didn't just make up as part of our marketing. (see OpenStand.org)

So first off, it may be useful to note that when looking for a universal, interoperable standard for text chat, the armed forces around the world went for XMPP. This based on its reliability and handling of low bandwidth situations (much, much, MUCH lower bandwidth than mobile). So when you say XMPP doesn't support your notion of an MVP, it's worth noting it really is just that - your notion. That doesn't make you wrong, but it may be that your focus is different from other people's.

And as a second point, when considering writing something on the web, you (thankfully) don't have to implement JavaScript. Or HTML. Or CSS. Or, thank heavens, HTTP. Do you even know what happens in HTTP? Clearly not, otherwise you'd be a gibbering wreck - it's about as badly designed as a protocol can be and still work. And don't get me started on TCP; nobody gets that right.

So really, if you're implementing something using XMPP, please don't leap in at the low level - just use one of the many libraries for your chosen platforms.

Next: Unique message identifiers are an interesting topic, and you've referenced several XEPs that don't address the problem. One issue is that what's needed varies in different situations - generally, a short-term scoped identifier is good enough, but it's fair to say that's not always the case. The tuple of <from,to,id> is good enough for this, though clients aren't mandated to send an identifier. It's not even clear to me that we need global stable persistent single-valued identifiers, but despite there being no use-case explained to me where this is required, I'm aware it's a popular solution some undefined problem.

Worth noting, too, that XEP-0198 and XEP-0184, though often confused as addressing the same problem, actually address entirely different ones. '198 addresses link reliability, tackling the two generals problem. '184 addresses end to end acks. You end up with acks in both, but 184's can be out of order, and 198 allows session resumption and retransmissions, which addresses the same cases you identify with vector clocks between domains. Your library should handle all this for you, and just note if a message is lost.

Carbons you have entirely wrong, sorry. It does not involve "re-sending your messages back to the server", it just allows non-priority clients to get copies of all messages. The only oddity is that the copies are just that - they're not duplicates. This allows the client being copied to identify that the messages are going to a different device. Which device is usually based on "priority", though different servers can do things slightly differently. Despite this, it doesn't have anything to do with message traffic. And it's not random.

However, clients typically respond to the device which sent the message - but if the sender has moved devices, they'll get a carbon. And yes, there is a massive "if supported" right there, but there are desktop clients that do.

In fact, we don't handle the "every device has the same view of the universe" concept - you're right - we handle the "every device knows the state of every other device". It's functionally identical, but a slightly richer view.

So, let's rewind.

The XSF doesn't write software, though there are many libraries out there. We're working on Push, mostly to cover Apple's platform where long-running sessions are prohibited - this despite the fact that on Android, clients like Yaxim or Conversations don't excessively use power.

The XSF also doesn't do the levels of social media advertising that Matrix does. This is becoming increasingly problematic for us as a community, because Matrix are explicitly targetting XMPP with things that are simply untrue.

Most of the problems you identify are actually solved at the protocol layer. They really are. And it's not as bad a solution as you imply, either - in fact, the solutions to multi-device messaging and archiving are pretty good, and more based on "every device knows about the other devices" rather than "every device is treated identically", which is interesting in and of itself.

Oh, and when I said low bandwidth? XMPP is routinely used over 2400bps SATCOM links with 30s latency, but it'll run lower. Try 9 bits per second, half-duplex, with 2 minutes link turnaround.

[Arathorn]

heavenlyhash: I think you struck a nerve ;)

Dave: yes, the Matrix 3rd party ID to Matrix-ID mapping service is currently logically centralised. The good news is that the service is entirely optional and you don't have to use it, and indeed in practice today nobody does: as of today everyone uses Matrix IDs to talk to each other. Matrix IDs themselves are completely decentralised, just like Jabber IDs, and are indeed fully partitioned by domain. I'm afraid you're the one shamelessly spreading the FUD here :D That said, if you have suggestions on how to decentralise a 3rd party ID to JID or Matrix ID mapping database, we'd love to collaborate on it, given the user discovery problem is an open issue for both XMPP and Matrix.

In terms of whether XMPP and Matrix are competitors: personally, I don't think of it that way. They have utterly different semantics and features. Matrix is fundamentally a decentralised object database with pubsub; XMPP is an extensible message-passing system. It might be worthwhile trying to play nice together rather than flaming each other - for instance, one of the folks in the Matrix community is writing an XMPP<->Matrix bridge that could benefit both ecosystems.

I'd also point out that there are loads of chat use cases where XMPP is currently miles better than Matrix - Matrix is inherently a heavier protocol due to all the state synchronisation, and if you want to just chuck messages around the place then I'm sure there are super-speedy mature XMPP servers that let you do so.

The idea that Matrix is branding for a commercial outfit is pretty ridiculous for anyone who's actually spoken to or worked with us :) I guess we're lucky that we've got corporate sponsorship to let us run around and try to tell folks that we exist, but the whole project is entirely transparent and open in every sense of the word I can imagine. Please stop hating and consider embracing the existence of a complementary open initiative rather than getting all defensive all the time...

[davecridland]

OK, so your website claims that identity mapping is an integral key feature, so if I'm spreading FUD I apologise for repeating what's on your website.

As to whether XMPP and Matrix are competitors, you make direct comparisons between them (with incorrect assertions that have been pointed out to you before), and since you attack XMPP constantly on Twitter et al, I take it you think it is a threat.

Specifically, from your website's FAQ: - There's around four or five XMPP/Web implementations allowing you to easily speak XMPP from a browser, but the two most popular with web developers seem to be stanza.io and xmpp-ftw - the latter is JSON objects via HTTP/Websocket. - Most server implementations cluster, so a chatroom is highly available across multiple physical servers within the same domain. This latter restriction can be avoided using FMUC. - Hedging your comments with "(without extensions)" is crass, since there are many extensions that have very wide support. - Talking about a minimal baseline is at best ignorant, at worst deliberately misleading. Chatrooms aren't in our baseline, but that doesn't mean we should ignore them. - "Not particularly suited to mobile". We've actively worked on push recently, but on Android it's really optional. As for bandwidth efficiency, you use HTTP, so that's laughable - as noted above, stanzas go across HF radio in their native format just fine.

Yes, you can chuck messages around fast in XMPP. You can also chuck messages through pubsub systems very fast in XMPP.

So Matrix is a brand with no legal entity behind it, where the people operating it and controlling the specification seem to be entirely employed by the same company who owns the domain and sponsors an extensive publicity campaign? I shall never suggest it's merely branding for a commercial outfit again, my deepest apologies for having clearly misunderstood the situation.

As for getting defensive, I admit that too, but it's quite common when being attacked.

[Arathorn]

I'm pretty sure we don't claim identity mapping anywhere as an integral key feature; if we do, it's a mistake. We always decoupled the identity mapping layer from the core messaging layer to avoid the Hard Problem of decentralised identity mapping blocking the core protocol, and we haven't even specified it yet (that chapter of the spec is blank).

In terms of whether we've been doing some kind of Evil Disinformation Campaign against XMPP, looking at https://twitter.com/search?q=%40matrixdotorg%20xmpp&src=typd... if anything it seems like we're pretty supportive of XMPP: "@usetalky stanza.io looks lovely.", "@Nyssen11 converse looks pretty. we (or someone) will write an xmpp s2s <--> Matrix bridge soon, i'm sure.", "For sure XMPP is doing cool new stuff too - FMUC, XMPP-FTW, Buddycloud etc." etc etc. We even promote XMPP alongside Matrix: "Metadata privacy & federation with legacy networks are mutex. If you want metadata privacy @GNUnet ftw. For fed, Matrix or XMPP?"

The only valid beef I'm seeing is https://twitter.com/ckoehncke/status/588341851360518144/phot... which missed that XSF had published a Push XEP a few weeks earlier (sorry!), and ".@rikardlinde @davewiner Matrix is pure HTTP & decentralised convo history: no single silo/point of control. Jabber MUCs = single chatserver", which was admittedly ambiguous and misleading thanks to the 160 char limit and I subsequently clarified; the intention was to point out that MUCs = single logical chatserver locked to a single domain (ignoring FMUCs).

In terms of the FAQ - as per our current twitter convo I'm updating it in realtime to incorporate your POV.

In terms of whether we are a Nefarious Corporate Conspiracy: We're in the process currently of splitting out Matrix.org as an independent UK Limited By Guarantee company with not-for-profit statutes of incorporation to act as the neutral guardian of the Matrix standard. I guess this is a bit like how IBM split off the Eclipse brand into being the proper Eclipse Foundation. So yes, from my pov we're not 'branding for a commercial outfit', given 100% of the IP for the project is permissive-licensed opensource and the project is non-profit rather than commercial. Meanwhile, increasing amounts of the Matrix ecosystem are being contributed by the community (like the aforementioned XMPP<->Matrix bridge ;) Anyway, this will all be clearer once we have our separate legal entity - apology accepted for misunderstanding the situation :D

[davecridland]

http://matrix.org/docs/spec/#identity

""" Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix users. A Matrix "Identity" describes both the user ID and any other existing IDs from third party namespaces linked to their account. Matrix users can link third-party IDs (3PIDs) such as email addresses, social network accounts and phone numbers to their user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping can then be used by Matrix users in order to discover the MXIDs of their contacts. In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID and persist and replicate the mappings. Usage of an IS is not required in order for a client application to be part of the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs. """

OK, so aside from that last paragraph, that reads very integral. If it's just a case of "you have an identity at a domain" by default, and the 3PIDs are all an extension^Woptional-feature-that's-part-of-the-baseline, then what's the "strong identity system" you claim XMPP is lacking?

Actually if you read through that Twitter search, you see a bunch of XMPP folk getting annoyed at you for "half-truths, as usual", you referring to XMPP as a failure, you claiming that only the baseline counts, you claiming that MUC - universally and interoperably supported in every server (and every client that wants it) - is fragmented.

If this is you being supportive, I'd hate to see your actual disinformation campaigns.

Now, if you actually want a constructive conversation, that's great, but trolling just isn't the way to do that.

If you'd like a case where I suspect that Matrix models better, it's that ad-hoc, "ungoverned" multiparty chats work better in Matrix than XMPP.

That's because XMPP's multi-user chat model is (intentionally) based around IRC-channel-like models, where there's a single identity and authority. As I understand Matrix (and I don't claim expertise here), Matrix instead models a multi-party chat as simply a conversation involving multiple parties.

[Arathorn]

As the spec says, the ID service is optional. We haven't even bothered speccing it because the current logically-centralised thing is placeholder until we find a good decentralised solution. "Strong identity system" simply refers to mandating public key infrastructure both for servers and clients. For servers it's implemented via perspectives as per http://matrix.org/docs/spec/#retrieving-server-keys. For clients it's currently being defined for our new E2E stuff; we'll publish the key management APIs over the next few weeks.

I'm afraid I do believe that XMPP has failed in providing a ubiquitous (i.e. as ubiquitous as the web or email) open federated ecosystem for realtime comms, otherwise we wouldn't have tried to build Matrix. Obviously we may fail too, but hey, might as well try. Sorry if you consider this opinion trolling :)

In terms of "only the baseline counts" or "MUC is fragmented", I'm either mis-communicating or you're misunderstanding. I do think XMPP suffers from too low a baseline of functionality (as do others on this thread seemingly), and I think that the various alternative group chat technologies on XMPP (e.g. MUC v FMUC v Buddycloud) between them cause fragmentation, in that conversations can get stuck in one protocol inaccessible to another (please correct me if I'm wrong).

And yes, Matrix models multi-user chat as 'simply' a distributed datastructure replicated over multiple servers with eventual consistency, using decentralised ACLs to maintain authority in the chat.