[akhong]

Wrapping Spark in Clojure:

https://github.com/zero-one-group/geni

I used to work at a tech giant, where the data team relies a lot on native Spark in Scala. I've always found the combination quite pleasant to work with. However, I did miss Python's faster startup time, dynamism and REPL, especially when doing data cleaning and exploration with no intention of putting it in production.

Now that I'm doing my own thing at a much smaller scale, I naturally gravitated towards Python's data stack, namely NumPy, Pandas, Sklearn and Dask. However, I found myself missing Spark's consistent SQL API and performance!

So yea, I've been wanting to use more Clojure for work and set up a Clojure shop. During the quarantine, I find myself having more time to do focused work. I thought this would be a good opportunity to convert some of the data wrangling stuff to Clojure!