[puzzle]

Other tools and features that don't exist outside:

- a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should.

- load balancing dashboards showing traffic between frontends and backends

- load balancer support for dynamic sharding

- gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!msg/chrom... (I think most of this is doable these days, but the initial setup requires a Linux system)

- Kubernetes: server-specific custom hyperlinks on dashboards (e.g. links to POD_IP:PORT/stats, /debug, etc. for each individual pod you are looking at)

- Kubernetes: multiple Docker images in the same container or pod. E.g. the first container could be your code, while the second one might be data or the JVM runtime, etc., without having to bundle them together or doing costly copies in init containers.

- Kubernetes: canaries and automatic rollbacks

[necubi]

> - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should.

Envoy can do this, via its shadowing feature. See the docs here: https://www.envoyproxy.io/docs/envoy/v1.6.0/api-v2/api/v2/ro....

[puzzle]

Great! I suspected it had to be in there, but it was hard to figure that through the docs. I didn't think of 'shadow' as a keyword, only 'mirror'. Thanks. One more reason to adopt Envoy and Istio.

[necubi]

Yeah, the docs are definitely an area of Envoy that could be improved. All the information is there, somewhere, but finding what you're looking for can be quite challenging.

[kodablah]

> Kubernetes: canaries and automatic rollbacks

Hot off the presses: https://cloudplatform.googleblog.com/2018/04/introducing-Kay.... Though you have to use Spinnaker.

[puzzle]

That's an external controller, which is what most people are doing themselves these days, reinventing the wheel each time. Borg has long had an automatic rollback feature on updates, tuned through a few settings on top of the health check machinery. I'm in the camp believing that a basic implementation should be built-in, since health checks are already there. An implementation of this was started, but it has stalled. External controllers should be still allowed, for more advanced scenarios like rolling back upon detection of regressions in latency and similar.

Also, setting up Spinnaker is pretty much as complicated as Kubernetes itself. :-)

[lobster_johnson]

How does Borg express updates? With Kubernetes only deployments have rollout/rollback, and you need something like Helm -- which is frankly not that great -- to handle groups of resources that are to be updated and versioned together, e.g. configmaps and services.

[puzzle]

AFAIR, you can only update jobs (collections of single containers) and allocs (pods inside of which jobs will schedule and run: you need to state that job J lives inside alloc A, otherwise it will run in an anonymous, implied alloc). There aren't other objects you can manipulate, actually.

Configmaps don't really exist, although something similar is achieved with a job that has a second package holding just the data. This is why I think multi-image containers should be implemented, but also a reason why they haven't been yet: configmaps cover some use cases. When a job replica (task) gets updated or rolled back, both packages change in sync. On Kubernetes, you'd use version numbers in the configmap name (but you need to worry about garbage collecting unused ones).

Services live outside of Borg entirely. GSLB has its own push mechanisms and only consumes Borg's lists of containers that comprise a given Borg job.

From the paper:

> A user can change the properties of some or all of the tasks in a running job by pushing a new job configuration to Borg, and then instructing Borg to update the tasks to the new specification. This acts as a lightweight, non-atomic transaction that can easily be undone until it is closed (com- mitted). Updates are generally done in a rolling fashion, and a limit can be imposed on the number of task disruptions (reschedules or preemptions) an update causes; any changes that would cause more disruptions are skipped. Some task updates (e.g., pushing a new binary) will al- ways require the task to be restarted; some (e.g., increasing resource requirements or changing constraints) might make the task no longer fit on the machine, and cause it to be stopped and rescheduled; and some (e.g., changing priority) can always be done without restarting or moving the task.

[pm90]

> I'm in the camp believing that a basic implementation should be built-in, since health checks are already there.

Agreed 100%. Although... in my mind k8s is still pretty young, and this would definitely be a great feature to have by default in the future.

[philsnow]

> a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A

I would call that a "(live) traffic replayer" rather than a load balancer. "load balance" implies to me that the upstream traffic is divvied up among the downstream sinks, not that the upstream traffic gets copied to multiple downstreams.

[puzzle]

It's a load balancer, because, typically, there is more traffic than a single server can handle. That's why I mention pools: it sends traffic to one of {A1, A2, ...} and one of {B1, B2, ...}. Using your terminology, that instance of the load balancer will take the upstream traffic and divvy it up twice, among two separate groups of downstream pools. A load balancing tee proxy/LB is also useful where B is a new implementation that is supposed to handle more traffic. You could e.g. bring up 10 A replicas and 5 B replicas and compare their performance metrics. That setup is closer to reality than one instance of A and one of B (if you want to test soft stickiness, for example).

Anything at Google that does not support load balancing is doomed to melt fairly quickly.

[smarterclayton]

Image based volumes (second last to bullet) has long been blocked on the container runtime having a really clean way to enable and keep the container filesystems mounted. Definitely something I want to see fixed since otherwise you just end up doing hacky copies via emptydir.

[akhilcacharya]

Should you really be detailing the functionality of internal tools like this?

[puzzle]

Most of this is public (see the Slicer paper for dynamic sharding) or implemented in some other form by Envoy and other tools (dark launches, traffic visualization) or not that complicated (Kubernetes has manual rollbacks already and automating them is a natural extension). None of those are Spanner territory. Even LOAS is getting discussed and open sourced.

[QuercusMax]

None of this is particularly secret. Any intern who worked at Google would have access to all this info.

(Source: I'm a SWE at an Alphabet company)

[kyrra]

interns still sign NDAs.