[slt2021]

question about Arrow: the format seems to be not very space efficient.

I tried converting one of my parquet files from datalake from parquet to arrow and size difference is staggering. 20mb parquet -> 700mb arrow.

doesnt seem fit for datalake at all

[ayhanfuat]

Arrow is not really designed for storage though. See the "Parquet vs Arrow" section of this post (https://arrow.apache.org/blog/2022/10/05/arrow-parquet-encod...):

> Parquet and Arrow are complementary technologies, and they make some different design tradeoffs. In particular, Parquet is a storage format designed for maximum space efficiency, whereas Arrow is an in-memory format intended for operation by vectorized computational kernels.

> The major distinction is that Arrow provides O(1) random access lookups to any array index, whilst Parquet does not. In particular, Parquet uses dremel record shredding, variable length encoding schemes, and block compression to drastically reduce the data size, but these techniques come at the loss of performant random access lookups.

[slt2021]

the O(1) random access doesn't look like advantage at all, honestly.

if I load my parquet into memory - I will have O(1) random access to any row just as well.

plus, considering that Arrow recommends to work in chunks of 1000 rows per file, I am curious to learn exact tasks for which Arrow is optimizing for.

the only use case I can think of is transferring data between systems written in different languages/runtimes and doing zero serialization/deserialization, just send/receive memory buffers that are nicely mapped to dataframes.

[throwboatyface]

Into what format in memory? You can't just mmap into a parquet file and access whole records, it's a columnar format.

Arrow is absolutely designed for Interop between languages - often devs want to develop their core in something like Python, and the platform devs want to work on Scala. Arrow lets you write all the distributed system and data shuffling in Scala, but then users can access the same records in their user-defined code with minimal overhead

[alamb]

The following paper describes some of the tradeoffs between different formats

Deep Dive into Common Open Formats for Analytical DBMSs https://www.vldb.org/pvldb/vol16/p3044-liu.pdf

[iwd]

Do you have compression enabled? At least from Pandas, Parquet defaults to compressed and Arrow/Feather default to uncompressed. When I enable zstd compression, I get similar file sizes, and sometimes Arrow is smaller.

[slt2021]

I was just trying pandas native .to_parquet and .to_arrow() without any extra config knobs

[lomereiter]

Arrow format is not intended for storage, it's for in-memory data exchange between different libraries and languages.

[the_optimist]

Seems likely a memory management issue in the Arrow interface for the language you’re using.