Why is open census so expensive? Using the default ocgrpc views, we've exported the data to stack driver and datadog and have had to turn off both due to cost. Comparing to our hand rolled solution, it's exponentially higher. I believe it's due to how the latency histograms uses individual metric tags to count RPCs in each bucket. This is both expensive and really not usable in datadog.
Are you going to collaborate with folks working on Kubernetes or other projects to find a way to automatically expose z pages, e.g. to the Kubernetes dashboard or the GKE console pages? That's one of the first things that ex Googlers miss from Borg. Under Borg you have to create a magic file with an HTML fragment, but there were APIs to help with that.
Developers and devops folks can plumb those manually, of course, using an Ingress object, but that's tedious and unneeded friction.
Yep! The Java library has some basic z-page support, but adding high quality z-pages will be a big part of the OpenCensus agent and OpenCensus service that we're developing now. These can be used within Kubernetes environments or on plain VMs.
High quality z-pages are great, but if there's no easy and secure way for a developer to get to them, they become useless. Rather than their contents or generation, my question is about automatic discovery and secure, restricted proxying of the pages. I don't want to tell my colleagues to run ad-hoc kubectl commands for each service or pod they want to inspect.
Ah, got it. We've discussed exactly this (within Google) with some of the Kubernetes maintainers who have similar wishes to yourself. The most immediate z-pages related work is focused on making them better in the more general case, as they're extremely barebones at the moment and are specific to each language's library. After that we'll explore the native Kubernetes integrations that you're asking for.
Maybe a dumb question (and I certainly haven't read enough): Any plans or projects that decouple language support from exporter/backend support? Bummer to see a bit of fragmentation there, of course can understand architectural reasons for it.