Ask HN: Kibana, Grafana, Chatbots, SMS – How do you keep an eye on your product?

[BloodKnight9923]

I've been playing with ELK (Elasticsearch, Logstash, Kibana) stacks for the past couple months and that has given me great insights into what my product is doing in a clustered state in real time. That coupled with a few monitoring scripts and I've finally gotten past feeling helpless. I have a few people saying I should check out Grafana, and I've seen several paid services in this areas too (logentries for example).

I also use chatbots to notify me when things are going wrong, or say when a customer signs up.

How do you keep track of your products, keep monitoring costs low, and know what's going on?

[jmkni]

I've gotten really into Slack recently.

I just log everything ye old fashioned way (text files) and then post anything important/noteworthy to a couple of Slack channels.

It's really easy to build custom integrations and Slack is running on all of our devices so as long as we are all subscribed to the relevant channels I can target @channel in my integration and we know straight away if something bad is happening.

I've use the ELK stack on previous projects, it works really well and isn't that difficult to use (there are some great guides on Digital Ocean) and when I get the time I'm going to set it all up for my current project too.

The only thing is you have to host it somewhere, and it's more justifying the budget for that.

[BloodKnight9923]

I use slack for a lot of notifications too. We strapped a python chatbot on as an integration and gave the bot access to our product which made asking the bot what's happening really easy.

Chatops are catching on, and I'm intrigued. Thanks for the response!

[ghilston]

Hey, feel free to checkout my library on GitHub. Its goal is to simplify interacting with Slack API (which is amazing IMO) and let users focus on simply writing their bot.

Its still very much in development, but figured this thread was relevant enough for me to post a link.

https://github.com/GregHilston/Simple-Slack-Bot

[BloodKnight9923]

Hah. I went to college with you. Small world. Will check out your library, I've been doing a lot of work with chatbots over the last year, focused on Slack. I've only really been working with Java solutions though, so a python library is a nice change of pace.

Thanks for the link!

[ghilston]

Oh hey man! Please do, feel free to use it in its current state, but the dev branch will be merged to Master shortly and break current compatibility.

Its a sweet simple library to make slack bots real simple!

[man5quid]

Elastalert + Kibana + Slack is great if you use logstash/streamstash to aggregate logs internally. I also combine new relic and slack for incident management.

[BloodKnight9923]

Never heard of Elastalert, I'll have to check that out. By the name I'm guessing it's triggers off an elasticsearch server, which would cover infrastructure I already have - so that's nice.

Thanks for the suggestion!

[wampwamp]

For log aggregation I use Elasticsearch + Logstash + Kibana. For statistics I use Influxdb + Telegraf + Grafana. For service alerts I use Consul + Consul Alerts. Notifications get sent to Pager Duty and/or Slack from one of Kibana, Grafana, or Consul.

The Grafana graphs are especially fun to watch and Kibana has a nice logentries-esque plugin for searching and tailing logs.

It's quite the system to setup but once it's running everything else (i.e. the rest of your app) is much more manageable.

[BloodKnight9923]

I've been meaning to use grafana for more than homelab server monitoring.

I've always found it so satisfying to sit back after setting up these types of systems and to watch everything churn out metrics / logs. I'll check out the kibana plugin, I'm using that in production at the moment.

Thanks for the reply!

[atmosx]

If you're a lone developer (or a small team of developers) isn't a lot better to use something like CloudWatch instead of messing with ELK?

[wampwamp]

For small apps I would use Monit to watch the server processes combined with CloudWatch to watch the servers themselves.

I would investigate the TICK stack (influxdb) before ELK. TICK is great for metrics that are not stored in log files. tail + awk should be enough for managing log files if you only have a few servers.

[BloodKnight9923]

I'm only working with logs across 8 servers with a lot of cross talk and ELK covered the logging side brilliantly, but falls over on metrics from what I have seen so far. I'll check out the TICK stack, that sounds like it may be a welcome replacement.

Thanks for the suggestion!

[exodos]

Zabbix(zabbix-slack-alertscript) & StatusCake + Slack