[maxaf]

I won’t even mention the anecdata from my own workplace. I’ll only say that within my extended network people are frantically rewriting Scala bits in other languages. Those who are stuck with it (because of, say, extensive prior investment into Spark) are scrambling to come up with alternate solutions.

Put your ear to the ground, and you too will hear it.

[xvilka]

There is an ongoing work on creating Spark alternative in Rust[1][2]. Hopefully, that could help.

[1] https://github.com/rajasekarv/native_spark

[2] https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native...

[pkolaczk]

Will take years to get there, if they even get there at all. This is a single-man side project now. Also benchmarking something that has 1/100 of the original functionality and is not production ready is very unfair. It is much easier to get a specialized use case faster than a general purpose system.

[playing_colours]

He is not the first, Andy Grove worked on a similar thing last years, but he worked mostly alone, and without community you cannot take off such ambitious project: https://andygrove.io/2018/11/datafusion-2019/

[dominotw]

> rewriting Scala bits in other languages.

My employer is doing the same.