[farhanhubble]

I love how much is possible with the Observable framework and support for libraries like d3.js. However many data apps cannot precompute their outputs. For example, a pipeline that extracts text from documents based on what a user queries, cannot precompute the results and any visualizations must be updated every time. The best hack to accomplish this seems to be rebuilding the app on each update. Or is there another solution?

[recifs]

The code in a Framework can do whatever you want it to do—it can load data on demand, call an external API, etc. Precomputing data is only an option, not an obligation.

But even when you want things to be very interactive it is a good idea to minimize the data. Expose only the "rows and columns" that you need, and compress it as much as possible. This can be done in a data loader. For example, see the data app we deployed yesterday on hugging-face: its data loaders ingest a large source database (320 files totaling 200GB), and digests it into a single 8MB parquet file that we can then use on the page to "live query" 3 million newspaper titles and dates. https://huggingface.co/spaces/observablehq/fpdn

[RyanHamilton]

I took the query only what you need approach with PulseUI: https://www.timestored.com/pulse/ It emphasises not sending all data to the client. For real-time and very large data sets this is the only option.