[whoevercares]

I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet. Of course it didn’t exist 4 years ago, but would it cheaper to build?

[spott]

It sounds like a lot of the value here was in a more appropriate level of abstraction, not in the speed of the compute itself.

Ray allowed them to optimize elements that spark didn’t, and that was what improved performance, not that spark itself was slow.