[pantsforbirds]

Anyone have a good benchmark comparing DuckDb to Parquet/Avro/ORC etc.? Super curious to see how some of those workflows might compare. Obviously at scale its going to be different, but using a single parquet file/dataset as a db replacement isn't an uncommon thing in DS/ML work.

[mytherin]

Why compare DuckDB to Parquet when you can use DuckDB and Parquet [1] :)

[1] https://duckdb.org/2021/06/25/querying-parquet.html

[tomnipotent]

Does DuckDB also use a PAX-like format like Parquet? Without going into code, the best I could find with a little googlefu is the HyPer/Data Blocks paper - is this a relevant read?

[mytherin]

DuckDB's storage format has similar advantages as the Parquet storage format (e.g. individual columns can be read, partitions can be skipped, etc) but it is different because DuckDB's format is designed to do more than Parquet files.

Parquet files are intended to store data from a single table and they are intended to be written-once, where you write the file and then never change it again. If you want to change anything in a Parquet file you re-write the file.

DuckDB's storage format is intended to store an entire database (multiple tables, views, sequences, etc), and is intended to support ACID operations on those structures, such as insertions, updates, deletes, and even altering tables in the form of adding/removing columns or altering types of columns without rewriting the entire table or the entire file.

Tables are partitioned into row groups much like Parquet, but unlike Parquet the individual columns of those row groups are divided into fixed-size blocks so that individual columns can be fetched from disk. The fixed-size blocks ensure that the file will not suffer from fragmentation as the database is modified.

The storage is still a work in progress, and we are currently actively working on adding more support for compression and other goodies, as well as stabilizing the storage format so that we can maintain backwards compatibility between versions.

[tomnipotent]

I should have clarified table storage specifically, to gain intuition on what happens when data is copied between DuckDB and Arrow/Parquet. Was much faster to just look at the code bridging the two and back track from there. Thanks!