At the moment, the capabilities of technology simply don't permit the level of... generic-ness? that vendors like OTel -- and customers really -- want. If you try to push everything through a single pipeline, as a single kind of data, you necessarily lock yourself out of most of the value you should be getting. You need a heterogeneous stack, with specific tools for specific purposes.
Opinions vary but my experience is that Prometheus-style metrics are by far the most important thing to invest in, and deliver the majority of the "observability value" to the broadest set of architectures. Tracing systems like Lightstep can be super useful, too, but to deliver value they need a lot more end-to-end integration effort, and the cost of setting it up can very very easily outweigh the benefits it provides.
I've come to believe that logs are a trap. Everyone understands them intuitively, so they feel comfortable using them for basically everything, without thinking very critically about the ramifications. And even when logs are structured, they have no substantive schema, so there's no backpressure, so to speak, on usage. So the signal/noise ratio almost immediately goes negative. And they just occupy enormous amounts of time and memory to manage, process, analyze, etc. Although I'm sure it's not a universal truth, I've found that everything you might normally think to log is actually much better served by in-process request tracing. In general that means maintaining the log events of the most recent N requests for all of a set of application-defined categories. This is basically a real-time view of a system, with history proportional to, I dunno, rarity? of the request class. You don't ship these anywhere, you ask the applications directly. It seems weird to describe but it's just vastly more efficient to manage, and equally if not more useful for debug and triage.
Opinions vary but my experience is that Prometheus-style metrics are by far the most important thing to invest in, and deliver the majority of the "observability value" to the broadest set of architectures. Tracing systems like Lightstep can be super useful, too, but to deliver value they need a lot more end-to-end integration effort, and the cost of setting it up can very very easily outweigh the benefits it provides.
I've come to believe that logs are a trap. Everyone understands them intuitively, so they feel comfortable using them for basically everything, without thinking very critically about the ramifications. And even when logs are structured, they have no substantive schema, so there's no backpressure, so to speak, on usage. So the signal/noise ratio almost immediately goes negative. And they just occupy enormous amounts of time and memory to manage, process, analyze, etc. Although I'm sure it's not a universal truth, I've found that everything you might normally think to log is actually much better served by in-process request tracing. In general that means maintaining the log events of the most recent N requests for all of a set of application-defined categories. This is basically a real-time view of a system, with history proportional to, I dunno, rarity? of the request class. You don't ship these anywhere, you ask the applications directly. It seems weird to describe but it's just vastly more efficient to manage, and equally if not more useful for debug and triage.
YMMV.