[icebraining]

I know I sound like a fundamentalist, but how is anything like REST? It's an RPC protocol! Nothing against RPC, but it doesn't fit any of the REST constraints, except for the client-server; it's not stateless, it can't have middle-man caching and it doesn't really follow an uniform interface. It's nothing like REST (and as such it doesn't get its benefits).

[geoffschmidt]

Speaking as the protocol designer, the eventual intention for DDP is to model reactive data set publishing in a sufficiently semantic way that it can be cached by proxies. We didn't get there in pre1 and decided to not let that block Meteor 1.0, but it is the goal for the future when we have some post-1.0 breathing room (and when the Meteor community needs it.)

The idea behind DDP is that HTTP got us three huge benefits: (1) shared tooling (I can write a caching proxy, and anyone with a website can use it); (2) interchangeable parts (your client and your server can be built with totally different languages and frameworks); and (3) easy APIs (I can describe my site's REST-y API to you in a page or two). These days, a lot of sites are moving away from REST/HTTP and toward ad-hoc, custom publishing schemes that run over websockets or a HTTP long-polling transport that is emulating websockets. Since they use these ad-hoc protocols to move around data rather than something like REST, they lose (1), (2), and (3). But all of these ad-hoc protocols are basically isomorphic to each other. DDP is an attempt to nail this kernel of data publishing functionality down semantically to the point that you can get those three benefits back.

As an aside, what's the pressure that's pushing these apps away from REST? The two big ones are:

- They cache data locally and need to keep the cache fresh (say they are apps that run in your browser and never reload the page, or they are native mobile apps), and they don't want to poll for updates. And/or,

- They need to do joins and they care about latency. (When you are loading the news feed, you want to fetch the feed stories, the comments, and the userpic URLs in one round trip, not first request the stories, and then only once you have them request the comments.)

Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL), and many modern apps need to perform some form of latency compensation (predicting the outcome of a RPC and simulating it locally to update the client's display while waiting for the server's answer, but then reverting to the authoritative outcome chosen by the server.)

DDP attempts to address these needs and run over a stream transport like websockets, while recovering some of the benefits of standardization that made REST so successful.

[icebraining]

Thanks; I can perfectly understand the advantages of having a standard protocol, and if DDP becomes an effective standard for applications outside of Meteor, it'll be a great achievement. I just wish people were more aware of what REST really means.

DDP seems like a good technology for webapps, but the issue is that I find webapps a really uninspiring trend in web development; my vision for the web is much more data-oriented than the code silos we're all making. And I believe the lack of understanding of what REST means is preventing people from seeing a bigger picture.

Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL)

I've heard that a lot, and I don't doubt that it's true in many cases, but usually it seems more representative of a difficulty in the modeling process.

For example, a transferBalance call certainly doesn't map well to updating any object, but creating a new Transfer (and passing the URLs of both Accounts) would be perfectly RESTful, in my opinion.

[porker]

> but the issue is that I find webapps a really uninspiring trend in web development; my vision for the web is much more data-oriented than the code silos we're all making.

Can you tell me more about your vision?

[icebraining]

I'm sorry, I started a reply, but after two paragraphs of rambling I had to delete it all.

If you want, I can try to form a more cogent reply later, but in a short summary, it would be keeping the current trend of sending structured data, but restoring the principles that made the web great - decoupling code from data, connecting it across services and communities (turning Hypertext into Hyperdata) and aggressively adopting standard formats.

I see a web where I don't go to Hacker News; messages come to me from everywhere, to where I want them to. Where I don't open an issue on Github, but on my personal tracker, which federates with theirs. Where businesses work for consumers, responding to their demands, instead of pushing their crap on us.

This view is mostly based on a mix of the concepts such as the Semantic Web / Linked Data, VRM, the Intention Economy and other ideas.

[geoffschmidt]

I'm totally there with you, and my hope with DDP 1 was to make some progress on that first step: decoupling code from data, so that a standard DDP client can connect to any DDP server and pull data out of it.

In the future, I'm super excited about investigating ideas like DDP discovery (asking a DDP server what data and methods it has) and description (having a DDP server publish metadata in some standard format that explains the semantics of the other data that it's publishing in terms of microformats). I think the main place we differ is that I think that these formats should use something more like a relational model rather than a hypermedia model, or a formal knowledge representation model like RDF. Isomorphisms exist between these models, and used properly they are all equally powerful, so really this just a design decision about what will be easiest to implement and adopt. And of course I believe that it has to be server push (websockets) rather than poll (HTTP), and that it has to support server-side joins (doing 1 query rather than N+1 queries, and taking 1 roundtrip rather than 2, to retrieve your newsfeed stories plus the comments on them.) I can't tell from your comment whether you agree with that or not, but like I said, these are two of the main things that motivate DDP.

All of that said, I think that by far the hardest part of the puzzle is the product design and federation issues. In your federated Github and HN, the technical work is the first 10%, the next 90% is finding a user experience for it that makes sense to and is ergonomic for a large number of real users, and the final 90% (really should be more like 9000%) is convincing the industry (read, creating incentives to force the industry) to cooperate with the federation and to resolve all of the political squabbles about advertising, control of the data, licensing, and so on. I mean, Google pulled out of XMPP federation -- all practical evidence suggests that keeping these federation together is wicked hard.

So that's why we're just trying to build the first 10% of that first 10%, which has a very clear use case for building rich browser apps and mobile apps -- and hoping that we will be a stepstool that others such as yourself can stand on to build the full solution that everyone dreams of.

Anyway, if you are sympathetic to this line of argument, I hope you'll check out DDP and see what interesting discovery and self-description features could be built on top of it, or suggest how its semantics could be strengthened!