[candiddevmike]

Recently switched from their cloud service back to on-premise. The cloud version wasn't being updated and the entire setup experience left a lot to be desired with how you connect their on-premise grafana agent, especially if you aren't using their easy button deployment stuff. Also, billing for metrics is insane, as on any given day my metric load may vary between 5-7k or more. This caused some operational overhead as I was constantly tweaking scrapers to reduce useless metrics.

For $50/mo, you can self host everything easier, cheaper and with more control IMO.

[maccard]

> For $50/mo, you can self host everything easier, cheaper and with more control IMO.

Can you give an example as to how you could self host a grafana stack for $50/month? On AWS that buys you 4 cores, 8GB memory and 0 storage, and it's certainly not easier than clicking one button on the grafana website.

[m1keil]

We are running Grafana and Prometheus on a single t3.xlarge instance with 150GB gp3 EBS.

Excluding traffic, it costs ~ $100 USD per month.

We are doing 10 second scrapes and currently have roughly 141k active time series. In Grafana Cloud it would cost...

15000 metrics for free. 126000/1000 * $8 = $882

Now here's the real kicker.. the pricing Grafana puts on their website are assuming 60 second scrape interval (1 data point/minute or DPM). If you are doing 6 DPM, that's $8 * 6 per 1000 time series!

So final bill.. drum rolls

126000/1000 * $8 * 6 = $6048

Yes. That's a 60x.

Now, sure, we don't get the scale, the backups, the SLA.. but we can live without it. And when Prometheus will start acting slowly, we will just bump it to t3.2xl, or spend some time and filter out some of the noisy metrics we might have around.

Btw, if you try to find any information about what is a "time series" or a "metric" on the Grafana's pricing page, good luck.

https://grafana.com/docs/grafana-cloud/metrics-control-usage...

[maccard]

> Excluding traffic, it costs ~ $100 USD per month

I don't doubt that that's affordable, or cost competitive to AWS, but thats' about as cheap as you can do it, _and_ that's not including traffic. It's pretty much impossible to half that bill.

[m1keil]

I excluded the traffic because the price is basically 0. This is internal traffic and a bunch of HTTP requests. It doesn't cost us $3000 a month.

[alexjplant]

There are Helm charts available for all Grafana products so if you already run a Kubernetes cluster and have spare capacity you can just throw it up there. Loki supports shipping logs to GCS/S3 natively and Prometheus can use Cortex (also available as a Helm chart) to do the same. Once you throw Grafana behind SSO and implement a backup cronjob you're done until you reach scale and have to start deploying/scaling individual components separately.

I implemented most of the above using Terraform on a managed DigitalOcean cluster on a Saturday a few months back; it wasn't super-hard. Alternatively you could rent a few VPSes someplace and use k3s or similar to get an unmanaged cluster.

[westurner]

Suggestions for organizing a Helm + Terraform [+ k3s/k3d/MicroShift] provisioning and monitoring git repo with CI for job accounting? (without Ansible & AWX, which I'd create a role with for this too)

- [ ] ENH,BLD: A cookiecutter for this would be cool

[krnlpnc]

> $50/month? On AWS that buys you 4 cores, 8GB memory and 0 storage

Self-hosting on AWS is kind of counterproductive. Look into "cloud" metal servers and the money will go much further.

[jrwr]

Two low end Hetzner/OVH Boxes for redundancy should do the trick

[jjeaff]

That's why AWS charges so much for outgoing traffic.