I don’t think your ideas are random nor out of place. Durable data accessible by different clients is an important property that bleeds into the data science realm. Apache Arrow seems to partially tackle this problem by moving the data to shared memory that can be accessed by different runtimes like Python and R.
There are a couple of problems I see with SQLite used as a common data format:
1. SQLite is a state machine that only stores the most recent version of mutable (binary) tables so change control is non-trivial.
2. Matrices/Tensors are not naturally stored in relational tables.
There are a couple of problems I see with SQLite used as a common data format:
1. SQLite is a state machine that only stores the most recent version of mutable (binary) tables so change control is non-trivial.
2. Matrices/Tensors are not naturally stored in relational tables.