|
|
|
|
|
|
by dekhn
687 days ago
|
|
|
Speaking as a distributed computing nerd, Ray is definitely one of the more interesting and exciting frameworks I've seen in a while. It's one of those systems where reading the manual, I can see that I'm not going to have to learn anything new, because the mental model resembles so many distributed systems I've worked with before (I dunno about anybody else, but tensorflow is an example of a distributed system that forced me to forget basically everything I knew before I could be even remotely productive in it). Unclear if it's in the best interests of anyscale to promote Ray as a general purpose cluster productivity tool, even if it's good at that more general use case. |
|
|
Our intention from the start was for Ray to be general purpose. And the core Ray APIs are quite general (basically just scheduling a Python function somewhere in a cluster or instantiating a Python class as a process somewhere in the cluster).
We had AI use cases in mind from the start, since we were grad students in AI. But the generality has really been important since AI workloads encompass a huge variety of computational patterns (allreduce style communication patterns on GPUs for training, embarrassingly parallel data processing workloads on spot instances, and so on).