[jamesblonde]

We are seeing more and more specialized query engines. This is a query engine specialized for training pipelines. It is not general purpose - it is for providing batches of training data at workers. It uses Ray for parallelization. The kind of queries you need are random reads (to implement shuffling across epochs), arrow support (zero copy to Pandas DataFrames), and efficient checkpointing.

[nyrikki]

Some of what they are doing is simply what was lost due to the ubiquitous nature of the relational model.

The hierarchical model is applicable to many problems and actually in part why moving off mainframes is challenging because IMS is so much more efficient than the relational model for applications like airline tickets.

There have been several efforts to leverage object stores in the way they did that I am aware of but it was a hard sell.

The hierarchical model really only works for many to one relationships, and it's integrity model differs and is not as DRY.

There are lessons to learn here but it requires some relearning.

When you have a shopping cart, having data local to the server handling the transaction is also a benefit.

Codd's relational model has advantages, but has held back some efforts because we are just use to dealing with the painful parts that we often don't consider other options.

[HackerThemAll]

DuckDB is specialized in efficient storage and fast query for analytics (OLAP), using a columnar storage (in contrast to row storage, used by usual RDBMSes doing OLTP processing). It's nothing new, it's been there for couple decades already. But this "distributed" DuckDB can indeed be beneficial for training.

[auxten]

Data operations are increasingly happening near the GPU side to boost efficiency—especially for compute-heavy workflows. Talking about Arrow file processing and zero-copy queries on DataFrames, which are becoming crucial for modern data pipelines. I think another option worth considering is chdb, which supports these features and fits well with this shift. (author of chdb here)

[agilob]

I'm super impressed how much effort DeepSeek did and how much of it they opensourced.