Hacker News new | ask | show | jobs
by agibsonccc 3522 days ago
The YARN stuff is interesting. I would be curious to see something akin to: https://spark.apache.org/docs/1.6.1/running-on-mesos.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn...

My main area of interest with this is: We have a lot of Java native interface code we run. I don't want folks to have to worry about configuring library paths and the like. My support loads get messy quick the second "c code" comes in the picture.

The ability to hook in to k8s to spin up executors would be pretty neat. The Mesos containerizer has that beat right now.

1 of the main reasons I heavily prefer mesos is also dc/os. dcos package install thing is head and shoulders above just having a docker runtime.

As for yarn: A lot of my customers use yarn and don't know what k8s is. OR if they do have a cluster that uses docker, they call it a "docker cluster" and it's often separate. It's not as hard of a sell for me for the YARN folks to just say: "install my docker daemon as an RPM or cloudera parcel so you can run executors"

vs: "Install k8s in your docker cluster with all this extra stuff in there"

K8s for me is in this limbo of: Not integrated enough or too complex for embedding in commercial use.

You guys have a great story going. Especially on the UX side of things. I know a lot of folks in the k8s ecosystem, but many of them aren't focused on big data or the JVM. Mesos on the other hand was spark's parent project. The integration levels show there. I'll be keeping an eye on it :).

My inherent problem with a lot of this is it's still prototyping. A lot of this support is still very green field and I'm stoked you guys are working on it. If anything because more competition is always good.

1 comments

Regarding packages, you might be interested in https://github.com/kubernetes/charts

Disclosure: I work on the Kubernetes project at Google.

Thanks! I'll take a look around. There's a lot more than just "apps" at stake here, but again: I'll keep an eye on the progress. We also do a lot with jvm based microservices and the lightbend/play stack which already standardized on DC/OS with conductr. Datatstax enterprise is also of interest to us. A lot of this stuff is already "baked".

There were a lot of little things that made us pick the stack we did - a lot of this is highly specific to our use cases and customer base. We will likely have a k8s version at some point. When that time comes (or if enough customers ask for it) I'll re evaluate what we need on the stack.

Thanks for the links!