[tomhoule]

I checked recently, and the Rust implementation of arrow — parquet as well — is sadly still not usable on projects using a stable version of the compiler because it relies on specialization.

There are some Jira issues on this, but there doesn't seem to be a consensus on the way forward. Does someone have more information, is the general idea to wait for specialization to stabilise, or is there a plan, or even an intention, to stop relying on it?

[nemothekid]

Last I checked when I tried to the library last the blocker on stable was packed_simd, which provides vector instructions in Rust. I can imagine the arrow/datafusion-guy isn't too keen on dropping vectorized instructions as that would be letting up a huge advantage and I'm imagine it's used liberally throughout the code.

As for stablizing packed_simd, It's completely unclear to me when that will land in stable rust. I recently had a project where I just ended up calling out to C code to handle vectorization.

EDIT: According to, https://github.com/rust-lang/lang-team/issues/29, the effort looks abandonded/deprioritized, so it may be a long time before it sees stable rust

[burntsushi]

> I recently had a project where I just ended up calling out to C code to handle vectorization.

packed_simd provides a convenient platform independent API to some subset of common SIMD operations. Rust's standard library does have pretty much everything up through AVX2 on x86 stabilized though: https://doc.rust-lang.org/core/arch/index.html --- So if you need vectorization on x86, Rust should hopefully have you covered.

If you need other platforms or AVX-512 though, then yeah, using either unstable Rust, C or Assembly is required.

[lordsunland]

The main blocker is specialization as it's being used in several places including parquet and arrow crate. As this feature is unlikely to be stabilized, we'll need to replace it with something else but so far it is challenging.

[tomhoule]

Thanks for clarifying that the plan is to stop relying on it :) Is there a specific place/issue/ticket to discuss this? If time allows, I would be interested in helping out.

[lordsunland]

There is https://issues.apache.org/jira/browse/ARROW-6717 tracking moving to stable Rust, and https://issues.apache.org/jira/browse/ARROW-4678 is also relevant. Feel free to create a sub-task under the former to remove specialization :)