[nmoadev]

Depending on the scale of your actual systems, you might be surprised how far you can get with open source tools like Prometheus / Graphana and just not configuring them for huge scale.

You could for example just run one VM for all of your observability stuff, and stick these tools on it and store data to disk.

Alternatively, if you've got some money and you're systems are OK with outbound internet connections. SaaS monitoring solutions like NewRelic, Dynatrace, etc. are much more plug-and-play.

[gsck]

This. I've began to roll out prometheus counters to some of our internal services to have some visibility on them. There is nothing fancy about our Prometheus config, its the absolute bare minimum required running in a docker container (One of 3 docker containers in the entire business) on the same box as the Grafana panel.

Its not going to win any awards but it works well enough for our needs, while requiring 0 maintenance whatsoever.