[reactordev]

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch.

That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your org, it’s going to be extremely painful to switch. Choose otel if you’re a hybrid cloud or you have on premises stuff. If you’re on AWS, CloudWatch is a better option simply because they have the data. Dead simple tracing.

[FunnyLookinHat]

I think you're looking at OTel from a strictly infrastructure perspective - which Cloudwatch does effectively solve without any added effort. But OTel really begins to shine when you instrument your backends. Some languages (Node.js) have a whole slew of auto-instrumentation, giving you rich traces with spans detailing each step of the http request, every SQL query, and even usage of AWS services. Making those traces even more valuable is that they're linked across services.

We've frequently seen a slowdown or error at the top of our stack, and the teams are able to immediately pinpoint the problem as a downstream service. Not only that, they can see the specific issue in the downstream service almost immediately!

Once you get to that level of detail, having your infrastructure metrics pulled into your Otel provider does start to make some sense. If you observe a slowdown in a service, being able to see that the DB CPU is pegged at the same time is meaningful, etc.

[Edit - Typo!]

[makeavish]

Agree with you on this. OTel agents allows exporting all host/k8s metrics correlated with your logs and traces. Though exporting AWS service specific metrics with OTel is not easy. To solve this SigNoz has 1-Click AWS Integrations: https://signoz.io/blog/native-aws-integrations-with-autodisc...

Also SigNoz has native correlation between different signals out of the box.

PS: I am SigNoz Maintainer

[reactordev]

Not confusing anything. Yes you can meter your own applications, generate your own metrics, but most organizations start their observability journey with the hardware and latency metrics.

Otel provides a means to sugar any metric with labels and attributes which is great (until you have high cardinality) but there are still things that are at the infrastructure level that only CloudWatch knows of (on AWS). If you’re running K8s on your own hardware - Otel would be my first choice.

[elza_1111]

FYI for anyone reading, OTel does have great auto-instrumentation for Python, Java and .NET also

[mdaniel]

A child comment mentioned k8s but I also have been chomping at the bit to try out the eBPF hooks in https://github.com/pixie-io/pixie (or even https://github.com/coroot/coroot or https://github.com/parca-dev/parca ) all of which are Apache 2 licensed

The demo for https://github.com/draios/sysdig was also just amazing, but I don't have any idea what the storage requirements would be for leaving it running

[6r17]

I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ?

I wonder if there are any other adapters for trace injest instead of OTEL ?

[bbkane]

There are a few: I've played with https://uptrace.dev and https://openobserve.ai/ . OpenObserve is a single binary, so easy to set up

[mdaniel]

be cognizant of their licenses (AGPLv3), it matters in some shops

https://github.com/uptrace/uptrace/blob/v1.7.6/LICENSE

https://github.com/openobserve/openobserve/blob/v0.14.7/LICE...

[darkstar_16]

Jaeger collector perhaps but then you'd have to use the Jaeger UI. Signoz has a much nicer UI that feels more integrated but last I checked had annoying bugs in the UI like not keeping the time selection when I navigated between screens.

[6r17]

Definitely should look up the tech more ; i lazily commented as Signoz clearly state it ingest most than 50 different sources ;

[elza_1111]

yep, SigNoz is OpenTelemetry native. You can instrument your application with OpenTelemetry and send telemetry data direclty to signoz.

[elza_1111]

There are integrations that let you monitor your AWS resources also on SigNoz. That said, I personally think CloudWatch is painful in so many other ways as well,

Check this out, https://signoz.io/blog/6-silent-traps-inside-cloudWatch-that...