[SEJeff]

60T sadly isn't that much. We tried (and failed) to use spark on a several PB dataset and it failed miserably.

[placeybordeaux]

Do you have anymore details? What version?

[SEJeff]

I want to say it was the highest 1.6.x around Feb/March of this year with a few PB (a sample of the real dataset) over an infiniband network. It just broke miserably. Also, java was never terribly good when you want to speak native ibverbs as the jni stuff is just slow.