[jashkenas]

I've got no immediate plans to play around with any Backbone-on-the-client-and-server stuff ... the next web app library in the works is only semi-related:

The basic idea is that for many public-facing websites (think NYTimes, natch, or Airbnb's search result listings, for example), the usual Rails convention of "Hey, here comes a request, let me generate a page just for you", is fairly inappropriate. Lots of "publishing" applications will melt very quickly if Rails ever ends up serving dynamic requests. Instead, you cache everything, either on disk with Nginx, in Memcached, or in Varnish.

But you know when the data is changing -- when an article has been updated and republished ... or when you've done another load of the government dataset that's powering your visualization. Waiting for a user request to come in and then caching your response to that (while hoping that the thundering herd doesn't knock you over first) is backwards, right?

I think it would be fun to play around with a Node-based framework that is based around this inverted publishing model, instead of the usual serving one. The default would be to bake out static resources when data changes, and you'd want to automatically track all of the data flows and dependencies within the application. So when your user submits a change, or your cron picks up new data from the FEC, or when your editor hits "publish", all of the bits that need to be updated get generated right then.

It's only a small step from there to pushing down the updates to Backbone models for active users ... but one step at a time, right? No need to couple those things together.

ps. Kudos to you for reading the source. It's always enlightening: https://github.com/angular/angular.js/blob/master/src/ng/roo...

[Hupo]

>But you know when the data is changing -- when an article has been updated and republished ... or when you've done another load of the government dataset that's powering your visualization. Waiting for a user request to come in and then caching your response to that (while hoping that the thundering herd doesn't knock you over first) is backwards, right?

>I think it would be fun to play around with a Node-based framework that is based around this inverted publishing model, instead of the usual serving one. The default would be to bake out static resources when data changes, and you'd want to automatically track all of the data flows and dependencies within the application. So when your user submits a change, or your cron picks up new data from the FEC, or when your editor hits "publish", all of the bits that need to be updated get generated right then.

You mean most things don't already do this? I've been working on a personal blog engine with this as one of the core ideas (basically all static assets and pages are compiled on edit), and I thought it was a pretty obvious way to go about it. Looks like I'm indeed not the only one to think of it, but how "new" you present the idea as is a bit surprising to me.

[bkirkbri]

For simple problem domains (Blog, Mom and Pop store website, etc) it's trivial to pre-generate content. For larger content systems you can run into a more complicated dependency tree. Then you have the choice between keeping the dependency logic accurate vs regenerating the entire content set on any change.

It also turns out that content sets that change infrequently, but also unpredictably are a pain to cache. You can cache them for a short time (as long as stale content can be tolerated), but then you lose cache effectiveness. Or you can cache it forever with some sort of generation/versioned cache, but that doesn't interface with named, public resources very well. Telling your visitors and Google that it's yourdomain.com/v12345/pricing not yourdomain.com/v12344/pricing doesn't really fly.

I definitely concur with your surprise about it being novel though. I think that for many situations it's just easier to run extra boxes to handle the increased load of generating dynamic content on the fly over and over again. It's good for SuperMicro and AWS. It's not so good for the planet.

I'm very excited to see Jeremy's approach to addressing the problem.

[lloeki]

> "Hey, here comes a request, let me generate a page just for you"

In a blogging context, stuff like Wordpress where pages are generated per request, then cached to handle any form of serious load just rubs me the wrong way... Such an infrastructure to display a few pages seems ludicrous.

> The default would be to bake out static resources when data changes, and you'd want to automatically track all of the data flows and dependencies within the application. So when your user submits a change, or your cron picks up new data from the FEC, or when your editor hits "publish", all of the bits that need to be updated get generated right then.

... so this is exactly what my WIP custom blog engine (ultimately meant to replace my posterous blog) looks like, initially composed of markdown source and makefiles, then ramped up to some rake tasks and a ruby library. An entity change (edit post, add comment...) should trigger generation of each page referencing it exactly once, and possibly immediately.

[rapind]

"But you know when the data is changing -- when an article has been updated and republished ... or when you've done another load of the government dataset that's powering your visualization. Waiting for a user request to come in and then caching your response to that (while hoping that the thundering herd doesn't knock you over first) is backwards, right?"

I'm not trying to pick a fight or anything, but it sounds like you're arguing against lazy loading?

Eager caching is very situational, and not something you want to do unless you can reasonably anticipate the thundering herd or have very few items or have unlimited resources to generate and store a complete cache.

I'm probably misunderstanding though.

[curveship]

Your comment reminds me of a CMS-like system I built over a decade ago based on plain old unix 'make'. 'make' tracked all the dependencies to determine what parts of the site needed to be updated when new content was added or updated. Content was authored unstyled in a simplified subset of html, and make ran it through an XSLT to do styling and aggregation, like to build indices. The whole thing worked very well, building over 2k pages in seconds. I still miss it!