【NOTE】Prometheus

This is a note taken when investigating Prometheus.

• https://prometheus.io/docs/
  • monitoring and alerting
  • built at SoundCloud
  • pull model via HTTP
  • for short-lived jobs, push gateway is available
  • similar services: Graphite, InfluxDB, OpenTSDB, Nagios, Sensu
  • OpenMetrics format (WIP)
  • resources: https://prometheus.io/docs/introduction/media/
  • prerecord expressions to improve performance = recording rules
  • PromQL for querying
• https://blog.alexellis.io/prometheus-monitoring/
  • cloud native time-series database and monitoring solution
  • time-series database with UI and querying language (PromQL)
  • stand alone process which provides /metrics endpoint = exporter
    • NodeExporter to send system data
  • /metrics endpoint is default
  • It's possible to set up side car process which has /metrics endpoint and exposes data of the application
  • Prometheus exposition format to expose your own metrics
• https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-ubuntu-14-04-server
  • There used to be a combination of Prometheus + PromDash
• https://dzone.com/articles/monitoring-with-prometheus
  • monitoring targets can be anything
  • prometheus server collects metrics from targets over HTTP
  • exporters to monitor third party systems
  • three components: server, targets, and visualisation layer
• https://www.katacoda.com/courses/prometheus/getting-started
  • Another dashboard by Weave Works: https://www.weave.works/features/prometheus-monitoring/
• AlertManager
  • https://pagertree.com/2017/12/01/prometheus-tutorial/
  • https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04
    • black box exporter
    • adding user to sudo group allows them to have privilege access (but no a root user)
      • Dedicated tutorial: https://www.digitalocean.com/community/tutorials/how-to-edit-the-sudoers-file-on-ubuntu-and-centos
    • prometheus server sends alerts to AlertManager based on rules
    • you can silence the alert and get it expired using amtool
• Prometheus + Grafana
  • https://dzone.com/articles/top-9-software-testing-wordpress-plugins-for-devel
• awesome Prometheus: https://github.com/roaldnefs/awesome-prometheus

TODO

• get familiar with Grafana
• setup mail server and send alert via email as well as Slack