[hnlmorg]

You'd expect them to be monitoring more than just the HTTP response codes from user requests for precisely this reason.

If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.

But effective monitoring is harder than people assume.

[dncornholio]

> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.

Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.

[hnlmorg]

No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why.

There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise.

There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing.

Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged.

Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear.

(this is just a super high level view of observability too)

[lokar]

Even a synthetic probe needs a few failures to trigger an alert.

You should not alert on cpu, ram, etc

[hnlmorg]

> Even a synthetic probe needs a few failures to trigger an alert.

It doesn't "need" that. That just how most people set it up because it’s an easy sane default that allows for network jitter without inexperienced engineers thinking about different conditions triggering different types of responses.

If you’re measuring internal APIs from an observablity solution that’s has nodes already inside you’re network enclave, then there is a strong argument for alerting early.

> You should not alert on cpu, ram, etc

That’s not true to say as an absolute statement. And a generalisation it heavily depends on the system your monitoring and how it behaves under pressure.

But in any case, I wasn’t suggesting CPU alerts were the end goal. I said:

> these types of metrics are generally bespoke to the type of application your monitoring.

Ie you’ll use metrics but those metrics will be highly specific.

The CPU examples were an illustration as to what a “metric” is (it might seem obvious but not everyone is an expert) but the point was HTTP response codes aren't the only types of metrics one should be capturing and watching.

[lokar]

Ah, yes, I misunderstood. And I have seen cases where a direct CPU alert makes sense, but 99 times out of 100 times I see it, it's nothing but trouble. Worse, I tend to see the cpu alert when there are no end to end synthetic alerts, 500 alerts, queue depth alerts, etc.

If your requests are fast and cheap, you can probe frequently relative to your goals, but often that's not really possible (think, long SQL queries, or scheduling a container/pod). There you need several datapoints, or possible fewer augmented with other signals.

[hnlmorg]

Yeah very true.

Talking about long SQL queries, I quite like throwing CPU alerts on database servers. They'll be a low priority alert (ie no out of hours "pagers") so just something that goes into a slack channel. But they're a good indicator of when developers have poorly optimized SQL, or the DB schema is poorly defined (eg missing indexes), or the DB server itself is poorly sized.

This wouldn't be something you'd expect to need in production and definitely not something you'd rely on as a notice of a production outage. But it is an example of one of those 1% occasions where a CPU alert does add value to the overall observability of the application.

But this also ties into your excellent point about how you'd use CPU and other data points to build a picture of what's happening in your application.

[re-thc]

> But effective monitoring is harder than people assume.

Who says public status page equals internal monitoring.

They likely know faster than you. Whether they post it publicly is a different issue (hint: SLA penalties, news impacting stock etc)

[hnlmorg]

I never mentioned anything about status pages.

Are you sure you’re replying to the right comment?

[re-thc]

> I never mentioned anything about status pages.

For context, the parent comment you replied to started with status page.

Then are you talking about internal leaks or just guessing? Otherwise besides what's public how do you know they don't know?

[hnlmorg]

It was two comments prior to mine that mentioned status pages.

Someone then replied about how it takes a bunch of HTTP response errors for problems to be alerted and thus I commented that application observability would consist of more than just waiting for users to hit errors.