[1egg0myegg0]

Have you thought about interfacing with DuckDB for out of core processing?

https://github.com/kimmolinna/DuckDB.jl

[jpsamaroo]

Firstly, I'll say that we already have work started to implement out-of-core directly in Dagger: https://github.com/JuliaParallel/Dagger.jl/pull/289.

With that PR in place, it should be possible to define a "storage device" which is backed by a database. I haven't had a chance to actually try this, since the PR still needs quite some work and testing, but it's definitely something on my radar!