[fnord123]

McKinney has been hard at work getting parquet and arrow support in pandas.

http://wesmckinney.com/blog/outlook-for-2017/

>Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day.

pyarrow; pyarrow.parquet (which uses parquet-cpp).

[dtjon]

Wow this is great. I've been working around the jvm to integrate sklearn and some spark jobs that produce Parquet. This is a huge relief

[sandGorgon]

Arrow doesn't do scikit - atleast last time I checked . Has it changed ?

[dtjon]

pyarrow has methods to convert to pandas, which scikit supports

http://pyarrow.readthedocs.io/en/latest/pandas.html

[sandGorgon]

No - this is not it. Scikit models need to be persisted. The only ways I have found is pickle or dill.

Take a look at this to understand what I mean . http://stackoverflow.com/questions/32757656/what-are-the-pit...