[twh270]

The most important thing to do IMO is to make it part of your acceptance criteria or definition of done for any story/feature. My observation is that observability gets ignored if it isn't baked into the culture.

In terms of implementation, if it isn't on a dashboard or doesn't fire off an alert, it doesn't exist. So put your telemetry and logs on a dashboard, and set up alerts based on your SLA's/SLO's.

Also, make it easy for developers to understand, create and edit dashboards/alerts. If it's hard to understand or hard to do, it won't get done.