[bhntr3]

Yeah, I'm trying to figure this out too. TensorFlow needs Yarn support. Ideally, Yarn would allocate resources and inform the processes of the various workers in the graph, etc. etc. I see that as the harder part. If you use mesos, then there is some preliminary support for that. https://github.com/tensorflow/tensorflow/issues/1996

Since TensorFlow has native dependencies on CUDA stuff for GPU support, I don't think there's much of a way to get around installing things on every machine. You might be able to package a python env without CUDA for spark to run using conda. Here's an interesting blog post about that: https://www.continuum.io/blog/developer-blog/conda-spark

But I'm not sure I see the point in running TensorFlow without GPU support. And if you're hoping to run GPU machines on an existing spark cluster and intelligently allocate the GPU stuff to the right machine. . . that's gonna be tough. Here's an interesting talk on that from the last spark summit: https://www.youtube.com/watch?v=k6IOWblLQK8&feature=youtu.be

Ultimately, you're probably better off just running your own gpu cluster strictly for your TensorFlow model on ephemeral AWS spot instances.

Or just use Google Cloud Machine Learning. That's what Google wants and expects you to do anyway. Borg is the Borg. You will be assimilated.

[nl]

TensorFlow without GPU support is very useful for inference.