Prometheus is a set of tools for monitoring, some of which are optional.
In it’s core, the Prometheus Server takes care of collecting and storing metrics in a highly performant time series database and makes them available for further processing (like querying them or sending alerts).
The Metrics are mostly scraped from so called exporters.
They are one of prometheus strengths, because the ones it brings are already powerful and there already exist a ton of exporters<\/a> from the community for all kinds of services, also it’s kind of simple to implement custom exporters.
Optional components are the Alertmanager<\/em>, the Pushgateway<\/em> and third party dashboards like Grafana<\/em>. <\/p>\n\n\n\n
What’s this article about?<\/h5>\n\n\n\n
I’m going to set up a Prometheus server, an alertmanager, a grafana server with an example dashboard and scrape some metrics to fiddle with promql and display them in a grafana dashboard.<\/p>\n\n\n\n
What’s needed?<\/h5>\n\n\n\n- 3 VMs with Debian Buster (bullseye should work too)<\/li><\/ul>\n\n\n\n
Let’s go… basic setup<\/h5>\n\n\n\n
I’ll name the three VMs “prometheus”, “grafana” and “alertmanager”,
make sure that prometheus is able to reach them all and grafana is able to connect to prometheus.
On prometheus install the prometheus package:<\/p>\n\n\n\n
root@prometheus:~# apt-get install prometheus <\/code><\/pre>\n\n\n\n On alertmanager install the prometheus-alertmanager package:<\/p>\n\n\n\n
root@alertmanager:~# apt-get install prometheus-alertmanager<\/code><\/pre>\n\n\n\nOn grafana install the grafana package.
With grafana it really is best to use the package they provide.<\/p>\n\n\n\n
apt-get install -y apt-transport-https software-properties-common wget gnupg\n\necho \"deb https:\/\/packages.grafana.com\/oss\/deb stable main\" >> \/etc\/apt\/sources.list.d\/grafana.list\n\nwget -q -O - https:\/\/packages.grafana.com\/gpg.key | apt-key add -\n\napt-get update\napt-get install grafana<\/code><\/pre>\n\n\n\nOn all hosts install the prometheus-node-exporter:<\/p>\n\n\n\n
apt-get install prometheus-node-exporter<\/code><\/pre>\n\n\n\nIf everything worked, then you should have a basic install by now and should be able to see the webinterfaces of the services.<\/p>\n\n\n\n
- prometheus.your.domain:9090<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/11\/Screenshot_20211113_132043.png\")
<\/figure>\n\n\n\n- alertmanager.your.domain:9093<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/11\/Screenshot_20211113_132307.png\")
<\/figure>\n\n\n\n- grafana.nik.local:3000<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/11\/Screenshot_20211113_132619.png\")
<\/figure>\n\n\n\nThe initial credentials are admin:admin.
<\/p>\n\n\n\n
Check if the exporters are working in general.
Therefore on any or all of the hosts do:<\/p>\n\n\n\n
curl localhost:9100\/metrics\ncurl oneofthehosts.your.domain:9100\/metrics<\/code><\/pre>\n\n\n\nYou should see lots of metrics like this, more on that later.<\/p>\n\n\n\n
# TYPE apt_upgrades_pending gauge\napt_upgrades_pending{arch=\"\",origin=\"\"} 0\n# HELP go_gc_duration_seconds A summary of the GC invocation durations.\n# TYPE go_gc_duration_seconds summary\ngo_gc_duration_seconds{quantile=\"0\"} 9.929e-06\ngo_gc_duration_seconds{quantile=\"0.25\"} 1.63e-05\ngo_gc_duration_seconds{quantile=\"0.5\"} 2.249e-05\ngo_gc_duration_seconds{quantile=\"0.75\"} 4.0521e-05\ngo_gc_duration_seconds{quantile=\"1\"} 0.00410069\n\nroot@prometheus:~# curl -s localhost:9100\/metrics | grep node_disk_written_bytes_total\n# HELP node_disk_written_bytes_total The total number of bytes written successfully.\n# TYPE node_disk_written_bytes_total counter\nnode_disk_written_bytes_total{device=\"sr0\"} 0\nnode_disk_written_bytes_total{device=\"vda\"} 8.2030592e+07<\/code><\/pre>\n\n\n\nBasic installation is finished and working.
Let’s check the…<\/p>\n\n\n\n
prometheus configuration<\/h5>\n\n\n\n
A quick overview:<\/em><\/strong><\/p>\n\n\n\nPrometheus server may be started with lots of arguments<\/em>.
There are four categories of them: <\/p>\n\n\n\n- config
- the config file<\/strong>, defaults to: \/etc\/prometheus\/prometheus.yaml.<\/li><\/ul><\/li><\/ul>\n\n\n\n
- storage
- parameters on where to store tsdb data and how to handle it<\/li><\/ul><\/li>
- web
- web paramaters, like the url under which prometheus is reachable, api endpoints, etc.<\/li><\/ul><\/li>
- query
- query parameters, like timeouts but also max-samples, etc.<\/li><\/ul><\/li><\/ul>\n\n\n\n
On a debian system, these parameters are defined in \/etc\/default\/prometheus and for this setup they are sufficient.
<\/p>\n\n\n\n
The prometheus.yaml can be split in the following sections:<\/p>\n\n\n\n
- global
- global<\/em> parameters<\/em> for all other sections like scrape and evaluation intervalls, etc., may be overwritten in the specific configurations.<\/li><\/ul><\/li>
- scrape_configs
- the actual job definitions on what to scrape, where and how<\/li><\/ul><\/li>
- alerting
- parameters for the alert-manager<\/li><\/ul><\/li>
- rule_files
- files that contain the recording and alerting rules <\/li><\/ul><\/li>
- remote_read + remote_write
- parameters for working with long term storage like thanos and\/or federation<\/li><\/ul><\/li><\/ul>\n\n\n\n
Rule files are periodically evaluated for changes and the prometheus-server itself may be SIGHUPed to gracefully reload it’s config (\/etc\/prometheus.yaml).
This can also be done via an api endpoint (\/reload) if enabled.<\/p>\n\n\n\n
\n\n\n\nLet’s build a useful config-file to scrape our hosts.<\/p>\n\n\n\n
This one, aptly named “node”, just scrapes the exporter on localhost (prometheus) and grafana.your.domain.
But, for the purpose of demonstration, I’ll override some parameters in job context, like scrape_interval, etc..
sample_limit is important here, because it prevents collecting too may samples, marking the scrape as failed. This may also prevent what is called “cardinality explosion” in Prometheus (more on this later).
The job config itself is static, which is OK if one has a defined set of metrics to check, which doesn’t change too often and get’s distributed and reloaded by some mechanism.
We’ll stick to that for now.<\/p>\n\n\n\n
scrape_configs:\n# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.\n - job_name: 'node'\n\n# Override the global default and scrape targets \n scrape_interval: 15s\n scrape_timeout: 10s\n sample_limit: 1000\n\n # metrics_path defaults to '\/metrics'\n # scheme defaults to 'http'.\n\n static_configs:\n - targets: ['localhost:9090']\n - targets: ['grafana:9100']\n<\/code><\/pre>\n\n\n\nNow, that there is an initial working config, one may access the metrics via the configured url of the prometheus server:<\/p>\n\n\n\n
http:\/\/prometheus.your.domain:9090\/targets<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211205_122931.png\")
<\/figure>\n\n\n\nOne may create a graph from those metrics now, by clicking on “Graph” and starting to type. For Example:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211205_123351.png\")
<\/figure>\n\n\n\nExecute it and you’ll get a list of Instances and Jobs with this metric.
Choose the one you are looking for (or all), copy it to the search bar and click on “Graph”.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211205_123644.png\")
<\/figure>\n\n\n\nGood, now that this is working, a few words on the scrape config and especially labels.
Prometheus uses labels for almost everything and has some powerful functions to manipulate and thus work with them.
Labels<\/strong> are the key\/value pairs associated with a certain metric.
In the example above, it’s instance=’grafana’<\/em> and job=’node’<\/em>, placed in {}, they are assigned to a metric.
Of course these are basic labels and one may add lots more.
That’s were one needs to be careful, because each unique combination of labels and metrics add a new time series in the database, leading to the aforementioned “cardinality explosion” eventually.
So, choose labels wisely and never ever use dynamic, unbounded labels, like user ids and such.
Read more on this topic: here<\/a>.<\/p>\n\n\n\nAlright, let’s check out how relabeling works:
Add a new label (this would also update an existing label foo with the value bar):<\/p>\n\n\n\n
- job_name: node\n static_configs:\n - targets: ['grafana:9100']\n - targets: ['localhost:9100']\n relabel_configs:\n - target_label: \"foo\"\n replacement: \"bar\"\n<\/code><\/pre>\n\n\n\nOne can also rename and drop Metrics which match regexes and also chain those rules, you may want to check out this excellent site<\/a> with examples.
However, don’t confuse relabel_configs<\/strong> with metric_relabel_configs<\/strong>.
metric_relabel_configs is applied after a metric was collected, but before it is written to storage, while relabel_configs is run before a scrape is performed.
You may use this to drop specific metrics until problems are fixed on a client, for example.
This one drops all the metrics for sidekiq_jobs_completion_seconds_bucket{job=”gitlab-sidekiq”}<\/em> from my gitlab server<\/a>.<\/p>\n\n\n\n- job_name: gitlab-sidekiq\n metric_relabel_configs:\n - source_labels: [ __name__ ]\n regex: sidekiq_jobs_completion_seconds_bucket.+\n action: drop<\/code><\/pre>\n\n\n\nYou could find such a problematic metric with this query, which will list the top 20 time series (check out this blog<\/a> on the topic):<\/p>\n\n\n\ntopk(20, count by (__name__, job)({__name__=~\".+\"}))<\/code><\/pre>\n\n\n\n
\n\n\n\nGrafana<\/h2>\n\n\n\n
Now that we have a rudimentary understanding of the config and are scraping the exporters on two of our hosts, let’s connect Grafana, so that we can paint nice Graphs with powerful promql.<\/p>\n\n\n\n
After login, find the configuration symbol on the left side and choose Data sources.
It is sufficient to enter the URL of your Prometheus server here.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211206_002415-1024x666.png\")
<\/figure>\n\n\n\nOf course, there is a ton of options and Grafana itself can be used to as a Frontend for many data sources like databases, Elasticsearch, and so on.
If you are configuring Grafana for an Organization, you might also want to check out the User- and Org Settings.
For the sake of this Tutorial, we are done with the configuration.
Let’s build some graphs.<\/p>\n\n\n\n
Add a new dashboard:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211206_095047-1024x464.png\")
<\/figure>\n\n\n\nklick on “add a new Panel”<\/p>\n\n\n\n
Our Data source “Prometheus” should be visible.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211206_095228-1024x81.png\")
<\/figure>\n\n\n\nCPU Graphs are always good, let’s add one for system and for User CPU Time.
<\/p>\n\n\n\n
avg(irate(node_cpu_seconds_total{mode=\"system\", job=\"node\",instance=\"grafana:9100\"}[5m]))\n\navg(irate(node_cpu_seconds_total{mode=\"user\", job=\"node\",instance=\"grafana:9100\"}[5m]))<\/code><\/pre>\n\n\n\nAs you type, Grafana will provide you with options for auto completion.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211206_100006-1024x612.png\")
<\/figure>\n\n\n\nDon’t forget to save and click Apply.<\/p>\n\n\n\n
What has been done here?
I started to type node_<\/em> which provides me with a list of all node_*<\/em> metrics and chose node_cpu_seconds_total<\/em>.
Then the labels<\/em> are added, within {}<\/em>.
If you know that you want mode<\/em>, just start typing it (you may also use the explorer (left side) first or just look through the the exporter output on host:9100 (respective port if you use another one).
Also we want it for the resources defined in job=node<\/em> and an specific instance<\/em>.
Then, the function irate<\/em> is applied.
As stated, prometheus brings some very good functions to deal with timeseries data, but there are so many of them, best read about it here<\/a>.
Rate and IRate though are used often (at least by me) and easily confused.
They are pretty similar, but work differently.
As a rule of thumb: irate reacts better to data changes while rate gives more of a trend (see link above on how they really work).
Both get us an “average” (calculated differently) per second rate of events.
For short term CPU Load, I’ll use irate [5m].
(See the picture below for the difference, when using rate<\/em>)<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211206_115901-1024x612.png\")
<\/figure>\n\n\n\nAt least (outside) I used the avg<\/em> aggregator function, to calculate an average over the cpu cores (there are two of them.
Read about aggregator functions here<\/a>.
Check out how it looks like without avg<\/em>.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/Screenshot_20211206_125343-1024x612.png\")
<\/figure>\n\n\n\nOf course there’s much more to Grafana.
For example: if you click on the “Preferences” Icon in the upper right of the new Dashboard, it is possible to define variables, which (e.g.) can replace mode, job and instance values, helping you to create dynamic and easy to navigate dashboards with drop down fields.
Also worth noticing is, that you don’t need to build every single dashboard yourself.
There are many well maintained and production ready dashboards on grafana.com\/dashboards<\/a>.<\/p>\n\n\n\n
\n\n\n\nAlertmanager<\/h2>\n\n\n\n
The Alertmanager is prometheus’ “Alert Router”.
It takes alerts, generated from Prometheus’ Alerting Rules through it’s API and converts them into notifications in all kinds of forms: Slack Messages, Emails, and so on.
In a production environment one would set up more than the one we are building for the sake of this article.
Alertmanager can be clustered and replicates it’s state throughout it’s nodes and also de-duplicates alerts.
How does it work?
In short:
Whenever an alert rule fires, prometheus sends and event to the alertmanager API (json) and it keeps doing this as long as the rule matches.
The alertmanager then dispatches these alerts into alert groups<\/em> (defined by several labels, as everything in prometheus, e.g. alertname), here grouping and de-duplication is done to avoid unnecessary alert spamming, also alerts may be categorized.
From here, each group will trigger the notification pipeline.
First is the Inhibition<\/em> Phase.
This basically allows for dependencies between alerts to mapped (think of: a switch fails and every rule for connected devices would start alerting, this won’t happen if configured correctly).
Mapping is done in the alertmanager.yaml and thus requires reloading.
Second is the Silencer<\/em> Phase.
It does what it’s name says, it silences Alarms, either by directly matching labels or by a regex (be careful with that, though).
Silences can be configured through the web interface, by clicking on Silences (http:\/\/alertmanager.your.domain:9093\/#\/silences).
If the alert was not handled by one of the previous phases, it get’s routed<\/em>.
Basically the alert is send to an endpoint, which has to be configured in alertmanager.yaml, route section.
Several enpoints are pre-configured with example configs.
The configuration:<\/em>
On prometheus.your.domain, add the Alertmanager section:<\/p>\n\n\n\n# Alertmanager configuration\nalerting:\n alertmanagers:\n - static_configs:\n - targets: \n - 'alertmanager.your.domain:9093'<\/code><\/pre>\n\n\n\nAlso, tell prometheus which rule files to evaluate:<\/p>\n\n\n\n
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.\nrule_files: \n - \"alerting_rules.yml\"<\/code><\/pre>\n\n\n\nCreate an alerting rule in alerting_rules.yaml.
This one is pretty straight forward.
Raise an Alert “NodeExporterDown” if “up” for our Job=node is not “1”, for one minute and label it as critical.
Also annotations such as links and other useful stuff may be defined.<\/p>\n\n\n\n
groups:\n- name: alerting_rules\n rules:\n\n - alert: NodeExporterDown\n expr: up{job=\"node\"} != 1\n for: 1m\n labels:\n severity: \"critical\"\n annotations:\n description: \"Node exporter {{ .Labels.instance }} is down.\"\n link: \"https:\/\/example.com\"<\/code><\/pre>\n\n\n\nNow we need to configure the alertmanager itself.
On alertmanager.your.domain edit \/etc\/prometheus\/alertmanager.yml.
As stated, there are several pre-configured endpoints, let’s just use email.<\/p>\n\n\n\n
global:\n smtp_smarthost: 'smtp.whatever.domain:25'\n smtp_from: 'alertmanager@your.domain'\n smtp_auth_username: 'user'\n smtp_auth_password: 'pass'\n smtp_require_tls: true<\/code><\/pre>\n\n\n\ndefine a route:<\/p>\n\n\n\n
route:\n receiver: operations\n group_by: ['alertname', 'job']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 3m<\/code><\/pre>\n\n\n\ndefine the receiver:<\/p>\n\n\n\n
- name: 'operations'\n email_configs:\n - to: 'your@ops-email.de'<\/code><\/pre>\n\n\n\nSo, now, if we stop the prometheus-node-exporter on grafana.your.domain, the rule get’s evaluated and prometheus enters a pending state for one minute to see if the service comes up again.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/node_exporter_down_pending-1024x303.png\")
<\/figure>\n\n\n\nIf it does not, the service is marked as down:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/node_exporter_down_status-1024x118.png\")
<\/figure>\n\n\n\nand the prometheus alert starts firing:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/node_exporter_down_firing-1024x287.png\")
<\/figure>\n\n\n\nWe can check for it on the alertmanager.your.domain web interface:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/node_exporter_down_alertmanager-1024x746.png\")
<\/figure>\n\n\n\nand also get an email:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.nikster.de\/wordpress\/wp-content\/uploads\/2021\/12\/node_exporter_down_mail.png\")
<\/figure>\n\n\n\nThat’s it. Prometheus, some basic rules, Grafana and a basic dashboard, and also Alertmanager and a basic alarming is up and running.
<\/p>\n\n\n\n
\n\n\n\nOf course there is much more to know, especially about alerting and recording rules, promql and dynamic configuration, as well as cloud configs and operators, long term storage and so on.
I’d recommend reading the excellent Infrastructure Monitoring with Prometheus<\/a> as well as the official documentation on the prometheus website<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"Infrastructure needs to be monitored and there exist several tools for this task, not at least because the term “monitoring” is rather fuzzy.However, two great tools for this task are graphite and Prometheus.Both have their pros and cons, like with graphite it is much simpler to keep data for long term analysis, while Prometheus shines … <\/p>\n
Let’s go… basic setup<\/h5>\n\n\n\n
I’ll name the three VMs “prometheus”, “grafana” and “alertmanager”,
make sure that prometheus is able to reach them all and grafana is able to connect to prometheus.
On prometheus install the prometheus package:<\/p>\n\n\n\n
root@prometheus:~# apt-get install prometheus <\/code><\/pre>\n\n\n\n On alertmanager install the prometheus-alertmanager package:<\/p>\n\n\n\n
root@alertmanager:~# apt-get install prometheus-alertmanager<\/code><\/pre>\n\n\n\nOn grafana install the grafana package.
With grafana it really is best to use the package they provide.<\/p>\n\n\n\n
apt-get install -y apt-transport-https software-properties-common wget gnupg\n\necho \"deb https:\/\/packages.grafana.com\/oss\/deb stable main\" >> \/etc\/apt\/sources.list.d\/grafana.list\n\nwget -q -O - https:\/\/packages.grafana.com\/gpg.key | apt-key add -\n\napt-get update\napt-get install grafana<\/code><\/pre>\n\n\n\nOn all hosts install the prometheus-node-exporter:<\/p>\n\n\n\n
apt-get install prometheus-node-exporter<\/code><\/pre>\n\n\n\nIf everything worked, then you should have a basic install by now and should be able to see the webinterfaces of the services.<\/p>\n\n\n\n
- prometheus.your.domain:9090<\/li><\/ul>\n\n\n\n
<\/figure>\n\n\n\n- alertmanager.your.domain:9093<\/li><\/ul>\n\n\n\n
<\/figure>\n\n\n\n- grafana.nik.local:3000<\/li><\/ul>\n\n\n\n
<\/figure>\n\n\n\nThe initial credentials are admin:admin.
<\/p>\n\n\n\n
Check if the exporters are working in general.
Therefore on any or all of the hosts do:<\/p>\n\n\n\n
curl localhost:9100\/metrics\ncurl oneofthehosts.your.domain:9100\/metrics<\/code><\/pre>\n\n\n\nYou should see lots of metrics like this, more on that later.<\/p>\n\n\n\n
# TYPE apt_upgrades_pending gauge\napt_upgrades_pending{arch=\"\",origin=\"\"} 0\n# HELP go_gc_duration_seconds A summary of the GC invocation durations.\n# TYPE go_gc_duration_seconds summary\ngo_gc_duration_seconds{quantile=\"0\"} 9.929e-06\ngo_gc_duration_seconds{quantile=\"0.25\"} 1.63e-05\ngo_gc_duration_seconds{quantile=\"0.5\"} 2.249e-05\ngo_gc_duration_seconds{quantile=\"0.75\"} 4.0521e-05\ngo_gc_duration_seconds{quantile=\"1\"} 0.00410069\n\nroot@prometheus:~# curl -s localhost:9100\/metrics | grep node_disk_written_bytes_total\n# HELP node_disk_written_bytes_total The total number of bytes written successfully.\n# TYPE node_disk_written_bytes_total counter\nnode_disk_written_bytes_total{device=\"sr0\"} 0\nnode_disk_written_bytes_total{device=\"vda\"} 8.2030592e+07<\/code><\/pre>\n\n\n\nBasic installation is finished and working.
Let’s check the…<\/p>\n\n\n\n
prometheus configuration<\/h5>\n\n\n\n
A quick overview:<\/em><\/strong><\/p>\n\n\n\nPrometheus server may be started with lots of arguments<\/em>.
There are four categories of them: <\/p>\n\n\n\n- config
- the config file<\/strong>, defaults to: \/etc\/prometheus\/prometheus.yaml.<\/li><\/ul><\/li><\/ul>\n\n\n\n
- storage
- parameters on where to store tsdb data and how to handle it<\/li><\/ul><\/li>
- web
- web paramaters, like the url under which prometheus is reachable, api endpoints, etc.<\/li><\/ul><\/li>
- query
- query parameters, like timeouts but also max-samples, etc.<\/li><\/ul><\/li><\/ul>\n\n\n\n
On a debian system, these parameters are defined in \/etc\/default\/prometheus and for this setup they are sufficient.
<\/p>\n\n\n\n
The prometheus.yaml can be split in the following sections:<\/p>\n\n\n\n
- global
- global<\/em> parameters<\/em> for all other sections like scrape and evaluation intervalls, etc., may be overwritten in the specific configurations.<\/li><\/ul><\/li>
- scrape_configs
- the actual job definitions on what to scrape, where and how<\/li><\/ul><\/li>
- alerting
- parameters for the alert-manager<\/li><\/ul><\/li>
- rule_files
- files that contain the recording and alerting rules <\/li><\/ul><\/li>
- remote_read + remote_write
- parameters for working with long term storage like thanos and\/or federation<\/li><\/ul><\/li><\/ul>\n\n\n\n
Rule files are periodically evaluated for changes and the prometheus-server itself may be SIGHUPed to gracefully reload it’s config (\/etc\/prometheus.yaml).
This can also be done via an api endpoint (\/reload) if enabled.<\/p>\n\n\n\n
\n\n\n\nLet’s build a useful config-file to scrape our hosts.<\/p>\n\n\n\n
This one, aptly named “node”, just scrapes the exporter on localhost (prometheus) and grafana.your.domain.
But, for the purpose of demonstration, I’ll override some parameters in job context, like scrape_interval, etc..
sample_limit is important here, because it prevents collecting too may samples, marking the scrape as failed. This may also prevent what is called “cardinality explosion” in Prometheus (more on this later).
The job config itself is static, which is OK if one has a defined set of metrics to check, which doesn’t change too often and get’s distributed and reloaded by some mechanism.
We’ll stick to that for now.<\/p>\n\n\n\n
scrape_configs:\n# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.\n - job_name: 'node'\n\n# Override the global default and scrape targets \n scrape_interval: 15s\n scrape_timeout: 10s\n sample_limit: 1000\n\n # metrics_path defaults to '\/metrics'\n # scheme defaults to 'http'.\n\n static_configs:\n - targets: ['localhost:9090']\n - targets: ['grafana:9100']\n<\/code><\/pre>\n\n\n\nNow, that there is an initial working config, one may access the metrics via the configured url of the prometheus server:<\/p>\n\n\n\n
http:\/\/prometheus.your.domain:9090\/targets<\/p>\n\n\n\n<\/figure>\n\n\n\nOne may create a graph from those metrics now, by clicking on “Graph” and starting to type. For Example:<\/p>\n\n\n\n<\/figure>\n\n\n\nExecute it and you’ll get a list of Instances and Jobs with this metric.
Choose the one you are looking for (or all), copy it to the search bar and click on “Graph”.<\/p>\n\n\n\n<\/figure>\n\n\n\nGood, now that this is working, a few words on the scrape config and especially labels.
Prometheus uses labels for almost everything and has some powerful functions to manipulate and thus work with them.
Labels<\/strong> are the key\/value pairs associated with a certain metric.
In the example above, it’s instance=’grafana’<\/em> and job=’node’<\/em>, placed in {}, they are assigned to a metric.
Of course these are basic labels and one may add lots more.
That’s were one needs to be careful, because each unique combination of labels and metrics add a new time series in the database, leading to the aforementioned “cardinality explosion” eventually.
So, choose labels wisely and never ever use dynamic, unbounded labels, like user ids and such.
Read more on this topic: here<\/a>.<\/p>\n\n\n\nAlright, let’s check out how relabeling works:
Add a new label (this would also update an existing label foo with the value bar):<\/p>\n\n\n\n
- job_name: node\n static_configs:\n - targets: ['grafana:9100']\n - targets: ['localhost:9100']\n relabel_configs:\n - target_label: \"foo\"\n replacement: \"bar\"\n<\/code><\/pre>\n\n\n\nOne can also rename and drop Metrics which match regexes and also chain those rules, you may want to check out this excellent site<\/a> with examples.
However, don’t confuse relabel_configs<\/strong> with metric_relabel_configs<\/strong>.
metric_relabel_configs is applied after a metric was collected, but before it is written to storage, while relabel_configs is run before a scrape is performed.
You may use this to drop specific metrics until problems are fixed on a client, for example.
This one drops all the metrics for sidekiq_jobs_completion_seconds_bucket{job=”gitlab-sidekiq”}<\/em> from my gitlab server<\/a>.<\/p>\n\n\n\n- job_name: gitlab-sidekiq\n metric_relabel_configs:\n - source_labels: [ __name__ ]\n regex: sidekiq_jobs_completion_seconds_bucket.+\n action: drop<\/code><\/pre>\n\n\n\nYou could find such a problematic metric with this query, which will list the top 20 time series (check out this blog<\/a> on the topic):<\/p>\n\n\n\ntopk(20, count by (__name__, job)({__name__=~\".+\"}))<\/code><\/pre>\n\n\n\n
\n\n\n\nGrafana<\/h2>\n\n\n\n
Now that we have a rudimentary understanding of the config and are scraping the exporters on two of our hosts, let’s connect Grafana, so that we can paint nice Graphs with powerful promql.<\/p>\n\n\n\n
After login, find the configuration symbol on the left side and choose Data sources.
It is sufficient to enter the URL of your Prometheus server here.<\/p>\n\n\n\n<\/figure>\n\n\n\nOf course, there is a ton of options and Grafana itself can be used to as a Frontend for many data sources like databases, Elasticsearch, and so on.
If you are configuring Grafana for an Organization, you might also want to check out the User- and Org Settings.
For the sake of this Tutorial, we are done with the configuration.
Let’s build some graphs.<\/p>\n\n\n\n
Add a new dashboard:<\/p>\n\n\n\n<\/figure>\n\n\n\nklick on “add a new Panel”<\/p>\n\n\n\n
Our Data source “Prometheus” should be visible.<\/p>\n\n\n\n<\/figure>\n\n\n\nCPU Graphs are always good, let’s add one for system and for User CPU Time.
<\/p>\n\n\n\n
avg(irate(node_cpu_seconds_total{mode=\"system\", job=\"node\",instance=\"grafana:9100\"}[5m]))\n\navg(irate(node_cpu_seconds_total{mode=\"user\", job=\"node\",instance=\"grafana:9100\"}[5m]))<\/code><\/pre>\n\n\n\nAs you type, Grafana will provide you with options for auto completion.<\/p>\n\n\n\n<\/figure>\n\n\n\nDon’t forget to save and click Apply.<\/p>\n\n\n\n
What has been done here?
I started to type node_<\/em> which provides me with a list of all node_*<\/em> metrics and chose node_cpu_seconds_total<\/em>.
Then the labels<\/em> are added, within {}<\/em>.
If you know that you want mode<\/em>, just start typing it (you may also use the explorer (left side) first or just look through the the exporter output on host:9100 (respective port if you use another one).
Also we want it for the resources defined in job=node<\/em> and an specific instance<\/em>.
Then, the function irate<\/em> is applied.
As stated, prometheus brings some very good functions to deal with timeseries data, but there are so many of them, best read about it here<\/a>.
Rate and IRate though are used often (at least by me) and easily confused.
They are pretty similar, but work differently.
As a rule of thumb: irate reacts better to data changes while rate gives more of a trend (see link above on how they really work).
Both get us an “average” (calculated differently) per second rate of events.
For short term CPU Load, I’ll use irate [5m].
(See the picture below for the difference, when using rate<\/em>)<\/p>\n\n\n\n<\/figure>\n\n\n\nAt least (outside) I used the avg<\/em> aggregator function, to calculate an average over the cpu cores (there are two of them.
Read about aggregator functions here<\/a>.
Check out how it looks like without avg<\/em>.<\/p>\n\n\n\n<\/figure>\n\n\n\nOf course there’s much more to Grafana.
For example: if you click on the “Preferences” Icon in the upper right of the new Dashboard, it is possible to define variables, which (e.g.) can replace mode, job and instance values, helping you to create dynamic and easy to navigate dashboards with drop down fields.
Also worth noticing is, that you don’t need to build every single dashboard yourself.
There are many well maintained and production ready dashboards on grafana.com\/dashboards<\/a>.<\/p>\n\n\n\n
\n\n\n\nAlertmanager<\/h2>\n\n\n\n
The Alertmanager is prometheus’ “Alert Router”.
It takes alerts, generated from Prometheus’ Alerting Rules through it’s API and converts them into notifications in all kinds of forms: Slack Messages, Emails, and so on.
In a production environment one would set up more than the one we are building for the sake of this article.
Alertmanager can be clustered and replicates it’s state throughout it’s nodes and also de-duplicates alerts.
How does it work?
In short:
Whenever an alert rule fires, prometheus sends and event to the alertmanager API (json) and it keeps doing this as long as the rule matches.
The alertmanager then dispatches these alerts into alert groups<\/em> (defined by several labels, as everything in prometheus, e.g. alertname), here grouping and de-duplication is done to avoid unnecessary alert spamming, also alerts may be categorized.
From here, each group will trigger the notification pipeline.
First is the Inhibition<\/em> Phase.
This basically allows for dependencies between alerts to mapped (think of: a switch fails and every rule for connected devices would start alerting, this won’t happen if configured correctly).
Mapping is done in the alertmanager.yaml and thus requires reloading.
Second is the Silencer<\/em> Phase.
It does what it’s name says, it silences Alarms, either by directly matching labels or by a regex (be careful with that, though).
Silences can be configured through the web interface, by clicking on Silences (http:\/\/alertmanager.your.domain:9093\/#\/silences).
If the alert was not handled by one of the previous phases, it get’s routed<\/em>.
Basically the alert is send to an endpoint, which has to be configured in alertmanager.yaml, route section.
Several enpoints are pre-configured with example configs.
The configuration:<\/em>
On prometheus.your.domain, add the Alertmanager section:<\/p>\n\n\n\n# Alertmanager configuration\nalerting:\n alertmanagers:\n - static_configs:\n - targets: \n - 'alertmanager.your.domain:9093'<\/code><\/pre>\n\n\n\nAlso, tell prometheus which rule files to evaluate:<\/p>\n\n\n\n
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.\nrule_files: \n - \"alerting_rules.yml\"<\/code><\/pre>\n\n\n\nCreate an alerting rule in alerting_rules.yaml.
This one is pretty straight forward.
Raise an Alert “NodeExporterDown” if “up” for our Job=node is not “1”, for one minute and label it as critical.
Also annotations such as links and other useful stuff may be defined.<\/p>\n\n\n\n
groups:\n- name: alerting_rules\n rules:\n\n - alert: NodeExporterDown\n expr: up{job=\"node\"} != 1\n for: 1m\n labels:\n severity: \"critical\"\n annotations:\n description: \"Node exporter {{ .Labels.instance }} is down.\"\n link: \"https:\/\/example.com\"<\/code><\/pre>\n\n\n\nNow we need to configure the alertmanager itself.
On alertmanager.your.domain edit \/etc\/prometheus\/alertmanager.yml.
As stated, there are several pre-configured endpoints, let’s just use email.<\/p>\n\n\n\n
global:\n smtp_smarthost: 'smtp.whatever.domain:25'\n smtp_from: 'alertmanager@your.domain'\n smtp_auth_username: 'user'\n smtp_auth_password: 'pass'\n smtp_require_tls: true<\/code><\/pre>\n\n\n\ndefine a route:<\/p>\n\n\n\n
route:\n receiver: operations\n group_by: ['alertname', 'job']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 3m<\/code><\/pre>\n\n\n\ndefine the receiver:<\/p>\n\n\n\n
- name: 'operations'\n email_configs:\n - to: 'your@ops-email.de'<\/code><\/pre>\n\n\n\nSo, now, if we stop the prometheus-node-exporter on grafana.your.domain, the rule get’s evaluated and prometheus enters a pending state for one minute to see if the service comes up again.<\/p>\n\n\n\n<\/figure>\n\n\n\nIf it does not, the service is marked as down:<\/p>\n\n\n\n<\/figure>\n\n\n\nand the prometheus alert starts firing:<\/p>\n\n\n\n<\/figure>\n\n\n\nWe can check for it on the alertmanager.your.domain web interface:<\/p>\n\n\n\n<\/figure>\n\n\n\nand also get an email:<\/p>\n\n\n\n<\/figure>\n\n\n\nThat’s it. Prometheus, some basic rules, Grafana and a basic dashboard, and also Alertmanager and a basic alarming is up and running.
<\/p>\n\n\n\n
\n\n\n\nOf course there is much more to know, especially about alerting and recording rules, promql and dynamic configuration, as well as cloud configs and operators, long term storage and so on.
I’d recommend reading the excellent Infrastructure Monitoring with Prometheus<\/a> as well as the official documentation on the prometheus website<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"Infrastructure needs to be monitored and there exist several tools for this task, not at least because the term “monitoring” is rather fuzzy.However, two great tools for this task are graphite and Prometheus.Both have their pros and cons, like with graphite it is much simpler to keep data for long term analysis, while Prometheus shines … <\/p>\n