[simonw]

There's an almost bewildering amount of interesting ideas buried in this.

Things like data loaders which are ANY script that can output data (as JSON or something else) to standard output. Markdown files with ```js blocks in that get executed. The reinvention of the core Observable notebook to avoid custom syntax.

This is really big.

[tophtucker]

Yeah data loaders are like a UNIX pipe to a reactive notebook cell (?). There needn’t be any question of “do the data loaders support this or that”; it doesn’t even have a concept of “supporting” beyond supporting stdout… Still thinking through how to understand it myself!!

[skybrian]

Data loaders seem like an interesting way to define a multi-language build system without having to write a makefile. Lots of build systems do this, but the boundaries between build steps often isn't as clean and uniform as having a single output per build step and relying on a file naming convention.

It's not truly reactive if you have to do a build to make anything happen. But maybe that doesn't matter, as long as it's reactive during development?

[tophtucker]

Yeah there are some open issues about more granular rebuilds and chaining data loaders. https://github.com/observablehq/framework/issues/638, https://github.com/observablehq/framework/issues/332

It’s kinda cool to think about the shearing layers of reactivity. Reactivity is what originally drew me to Observable. But the way notebooks have to be recomputed live for every viewer every time makes them feel silly for, like, a BI dashboard that changes daily at most. Like they only have one pace layer. Like they’re trying so hard to be _live_ that they can’t be _fast_! Idk. I guess even a chalkboard is reactive on the timescale of “someone noticing some information and telling it to someone who writes it down” lol.

[skybrian]

Another interesting example: since Go packages use minimum version selection, publishing a new version of a package doesn't do anything right away. Someone has to notice it (perhaps reading a release announcement) and bump the version number on a dependency. Then, after testing, they might release a new version, which again, doesn't do anything until projects downstream from them decide to upgrade.

That's deliberate, since they don't want packages to update their dependencies without testing them, and builds are supposed to be deterministic.

So it seems like for cross-project data dependencies, there's a tradeoff between deterministic results and getting the latest data? If one project depends on a JSON file from another project, and the JSON changes, when do you want or expect to see the change? There needs to be a version history for changes to the external JSON file to get a choice in the matter. (Perhaps it's cached locally.)

[tophtucker]

Haha yeah, that’s well put — “latest” depends on time, and something that depends on time is not deterministic, so those principles conflict. Currently a Framework project has the opposite priorities as Go and always gets the latest version of the data.

When I first started working at Observable I explained reactivity to my now-wife as “when one thing changes, other things that depend on it update automatically.” She was at an asset management job at the time working in Excel all day, which of course is also reactive in that sense. But she interpreted my statement differently; she said “Oh that’s terrible! If a number changes I want to know.” E.g. if the estimated budget for a project had ballooned, she wanted to hear about it so she could check why, ask other people about it, try to contain the damage, warn others, make other decisions differently, etc. “Automatic” would be harmful, like in your Go example. She was also thinking about coarser-grained data than I was — a single very impactful number, not a stream of mousemoves or pageloads or ad clicks or whatever.

[dleeftink]

Scripting with hot-reload?

[cxr]

The thing about Markdown code blocks is that they're meant to be read (and therefore rendered—as text). "Markdown files with ```js blocks in that get executed" is just <script>.