I think the main step forward is that it has an efficient means for calculating what views change when a new datum arrives. Which means that it could, in theory, hold a very large amount of views for relatively low overhead.
It's a surprisingly difficult problem. I wouldn't roll my own.
Never let HN haters (among whom I am frequently numbered, it ought to be noted) get you down.
After skimming a bit of the differential dataflow writing I am really impressed. This is deep computer science doing what it does best, which is to do much more with much less.
I’m glad to hear it! One of my favorite ways to view Materialize is as bringing differential dataflow to the masses. Differential dataflow is deep, elegant stuff, but it requires some serious CS chops to grok.
SQL is lacking in elegance but abundant in popularity. Building the SQL translation layer has been a fun exercise in bridging the two worlds.
The differential dataflow codebase was really polished and optimized last I saw - when they say "demand milliseconds" I think they put the effort into delivering that.
Additionally, given it will be Apache licensed in 4 years, I think it'd be good to ask "Can I wait?" before going all in
It's a surprisingly difficult problem. I wouldn't roll my own.