[Rapzid]

I'm still sitting on the sidelines waiting for the easy to install, better documented for AWS version. It's also a bit unclear as to why we are talking federation and master/slave in 2016; other systems are using raft and gossip protocols to build masterless management clusters..

Watching issues like https://github.com/kubernetes/kubernetes/issues/23478 , and https://github.com/kubernetes/kubernetes/issues/23174 .. I'm not super interested in "kicking the tires"; I'm evaluating replacing all our environment automation with a version built around Kubernetes. Easy-up scripts that hide a ton of nasty complexity won't do the trick.

Following the issues I'm getting the impression that too much effort is being put into CM style tools vs making the underlying components more friendly to setup and manage. Did anyone see how easy it is to get the new Docker orchestration running?

Then there is the AWS integration documentation.. I'm following the hidden aws_under_the_hood.md updates, but I'm still left with loads of questions; like how do I control the created ELB's configuration(cross zone load balancing, draining, timeouts,etc)?

I re-evaluate after every update and there are some really nice features being added in, but at the end of the day ECS is looking more and more the direction to go for us. Sure, it's lacking a ton of features compared to Kubernetes and it's nigh but impossible to get any sort of information about roadmaps out of Amazon... But it's very clear how it integrates with ELB and how to manage the configuration of every underlying service. It also doesn't require extra resources(service or human) to setup and manage the scheduler.

[thockingoog]

It's funny how words can be played. The Kubernetes "master" is a set of 1 or more machines that run the API server and associated control logic. This is exactly what systems like Docker swarm do, but they wrap it in terms like RAFT and gossip that make people weak in the knees. Kubernetes has RAFT in the form of the storage API (etcd). This is a model that has PROVEN to work well, and to scale well beyond what almost anyone will need.

"Federation" in this context is across clusters, which is not something other systems really do much of, yet. You certainly don't want to gossip on this layer.

"evaluating replacing" really does imply "kicking the tires". Put another way - how much energy are you willing to invest in the early stages of your evaluation? If a "real" cluster took 3 person-days to set up, but a "quick" cluster took 10 person-minutes, would you use the quick one for the initial eval? Feedback we have gotten repeatedly was "it is too hard to set up a real cluster when I don't even know if I want it".

There are a bunch of facets of streamlining that we're working on right now, but they are all serving the purposes of reducing initial investment and increasing transparency.

> how easy it is to get the new Docker orchestration running

This is exactly my point above. You don't think that their demos give you a fully operational, secured, optimized cluster with best-perf networking, storage, load-balancing etc, do you? Of course not. It sets up the "kick the tires" cluster.

As for AWS - it is something we will keep working on. We know our docs here are not great. We sure could use help tidying them up and making them better. We just BURIED is things to do.

Thanks for the feedback, truly.

[Rapzid]

I would consider "kicking the tires" actually running up a cluster and playing with it. One can also evaluate by reading documentation and others reports of issues to look for show-stopping problems. For instance, a couple releases ago there was not multi-AZ support. The word on the street at that time was to create multiple clusters and do higher level orchestration across them.. That was a no-go for us; no need to "kick the tires".

Whatever you may think of my level of knowledge or weak knees for consensus and gossip protocols, these problems(perceived or otherwise) with setup, documentation, and management seem pretty widely reported.

EDIT: I hope this doesn't sound too negative. Kubernetes IS getting better all the time. I only write this to give a perspective from somebody who would like use Kubernetes but has reason for pause. Our requirements are likely not standard; our internal bar for automation and ease of use is quite high. We essentially have an internal, hand-rolled, docker-based PaaS with support for ad-hoc environment creation(not just staging/prod). We would like to move away from holding the bag on our hand-rolled stuff and adopt a scheduler :) Deciding to pull the trigger on any scheduler though would be committing us to a rather large amount of integration effort to reach a parity that doesn't seem riddled with regressions over the current solution.

[justinsb]

This frustrates me greatly, only because I agree with you so vehemently :-) We have an open issue tracker where real decisions are made, and so engineers will argue about different approaches. Compare to alternatives, where you see demos that are double-acts of good-cop vs good-cop where apparently there are no trade-offs and everything is perfect. It isn't my experience that products where the debates are hidden are better; it is certainly easier to see the compromises when the debates are public.

So: there was a big discussion about whether a single k8s cluster should span multiple AZs (which shipped in 1.2), or whether we should allow the API to target multiple independent clusters (federation, the first version of which is shipping in 1.3). The core of the argument is that multi-zone is simpler for most users, but with only one control plane it is less reliable than a federation of totally independent clusters. Federation also brings other benefits, like solving the problem of running in clusters that are not in a single "datacenter" i.e. where you need to worry about non-uniform latency. I haven't seen anyone else make a serious attempt at solving this.

So, remember that the issue tracker is filled with the unvarnished discussions that come from true open source development. I think it is an asset for you, because you don't discover those things 3 months into using your chosen product; but it is definitely a liability for k8s, because we rely on you realizing this in your initial evaluation and weighting appropriately (the devil you know vs the devil you don't). I think k8s is likely much better than you think it is, and you should come talk to us on slack and make sure of that fact! It certainly sounds like you have an interesting use case that we'd like to hear about and consider.

But yes, our docs should be better!

[illumin8]

You might take a look at Rancher - it integrates and fully automates Kubernetes deployment, but personally, I find their Cattle scheduler is much easier to reason about, supports multi-AZ out of the box, and supports all of the features you would want (DNS-based service discovery, encrypted overlay networking, etc.)

Regarding the multi-AZ support issue - this is mostly because an EBS volume can only be attached to EC2 instances in the same AZ, and since Kubernetes has great support for persistent data volumes, you're pretty much limited to a single AZ if you're using persistent data volumes and want them to be remounted on a different instance in case of a failure. I think a more viable solution for persistent data volumes is to leverage EFS and use Convoy NFS to mount them. Now you have highly available, scalable, persistent data volumes, and you can stretch your cluster across multiple AZs.

[TheIronYuppie]

In this case, what you would do is set up two separate clusters, and spread an ELB across them. No federation required :)

Disclosure: I work at Google on Kubernetes

[illumin8]

But, if you have persistent EBS volumes, you wouldn't be able to mount them on the other cluster if you had a failure of an entire AZ.

[justinsb]

I do a lot of work for k8s on AWS (e.g. I'm to blame for the original under-the-hood document). I agree with most of what you've said we need to improve on, and we are working on it! We added a bunch of ELB configuration options in 1.3 and we'll likely complete the ELB feature set in 1.4 (it probably would have landed in 1.4 but for my tardiness in doing reviews). In addition, there's a well-known "trick" where you set up a service using a NodePort and then set up the ELB to point to that, if you want to go beyond what k8s offers (e.g. if you want to reuse an existing ELB etc). What our docs are lacking, we make up for with an excellent community (for AWS discussions #sig-aws on the k8s slack).

The debate between automation vs simplification is one that has gone on since k8s 1.0 and likely will continue to be had. But I think to an extent it is a false choice: I created a new k8s installation/ops tool (i.e. did work on the "automation" side), and out of that I got a fairly clear road-map for how we could simplify the installation dramatically in 1.4. In other words, some of the simplification you ask for comes by embedding the right pieces of the automation. k8s is open-source, so I have to persuade others of this approach, but I think that's a good thing, and I'd hope you'd join in that discussion also (e.g. #sig-cluster-lifecycle on the k8s slack).

[TheIronYuppie]

Can you say more about the complexity you're worried about? The projects you mention WILL land in 1.4 (which is about 90 days away), and thousands of companies are running huge production deployments on AWS, GCP, Azure, OpenStack and on-premises. Further, plug-ins already exist for Chef, Puppet, Salt and Ansible, if you'd like to use them.

To be clear, nothing is "masterless" - please go check out the production deployments for other container management solutions, they all require a separate control plane when running in production with a cluster of any reasonable (>64 nodes) size. FYI, it's a best practice when running a cluster of any size to separate the control plane.

To your direct question, with the other orchestration tools, how would you manage your ELB? Wouldn't you have your own management? They don't (to the best of my knowledge) do any sort of integration - not even the minimum level that Kubernetes does.

Disclosure: I work at Google on Kubernetes