Hacker News new | ask | show | jobs
by antisocial 2681 days ago
I didn't find Apache Arrow in this repo. I would like to learn more about your experience with using arrow, performance improvements and any lessons.
1 comments

I haven't looked into/keep up with Arrow much, but if I see fit, I can add more stuff about it :)!
I’ve given a very introductory talk about what Arrow “gives for free” when using the right kind of UDF. It’s more fun in person, but with the references at the end and the presenter notes I think you could get an idea of what you will want to mention quicker than having to look at it from scratch. It’s [here](https://github.com/rberenguel/pyspark-arrow-pandas), I hope you find it useful!
Oh awesome thanks for the resources! I will definitely see how i can incorporate it into my guide :).