31.10.2022

Monitoring Linux Services with Prometheus

server one
HOSTKEY
Rent dedicated and virtual servers with instant deployment in reliable TIER III class data centers in the Netherlands and the USA. Free protection against DDoS attacks included, and your server will be ready for work in as little as 15 minutes. 24/7 Customer Support.

Author: Senior Devops. Hostkey Lead Infrastructure Specialist Nikita Zubarev

In the last article we talked about how to set up metric collection and the display alerts to monitor services using Prometheus. However, during the process, the question arose of how to monitor individual services on Linux servers. For example, in one of our previous articles, we described the case of using Ovirt web consoles using Apache Guacamole. Now let's look at how to organize the monitoring of services on Linux machines.

Guacamole is a cross-platform remote desktop gateway that requires Apache Tomcat and Guacd to run.

Open the service created for node_exporter:

/usr/lib/systemd/system/node_exporter.service

[Unit]

Description=Prometheus exporter for machine metrics, written in Go with pluggable metric collectors.
Documentation=https://github.com/prometheus/node_exporter
After=network.target


[Service]

EnvironmentFile=-/etc/default/node_exporter
User=root
ExecStart=/usr/bin/node_exporter $NODE_EXPORTER_OPTS
Restart=on-failure
RestartSec=5s

[Install]

WantedBy=multi-user.target

by default, $NODE_EXPORTER_OPTS is loaded from the file:
# /etc/default/node_exporter


If necessary, we can either monitor individual services by adding the collector.systemd.unit-whitelist option:

#NODE_EXPORTER_OPTS="--collector.systemd --collector.systemd.unit-whitelist="(tomcat|guacd).service""

After restarting the service, we get the metric on node_exporter:

All you need to do is describe the rule for the alert:

- alert: guacd_service
	expr: node_systemd_unit_state{name="guacd.service",state="active"} == 0
	for: 1s
	annotations:
		summary: "Instance {{ $labels.instance }} is down"
		description: "{{ $labels.instance }} of job {{ $labels.job }} is down."
- alert: tomcat_service
	expr: node_systemd_unit_state{name="tomcat.service",state="active"} == 0
	for: 1s
	annotations:
		summary: "Instance {{ $labels.instance }} is down"
		description: "{{ $labels.instance }} of job {{ $labels.job }} is down."

An example of an alert on the Alertmanager dashboard in Grafana:

Conclusions

In this short article, we looked at how services are monitored on Linux machines.This solution allows you to monitor the work of any system server and greatly simplifies the administration of server hardware. In the next article, we will explain how you should not display the basic parameters that the exporter returns.

Rent dedicated and virtual servers with instant deployment in reliable TIER III class data centers in the Netherlands and the USA. Free protection against DDoS attacks included, and your server will be ready for work in as little as 15 minutes. 24/7 Customer Support.

Other articles

11.11.2022

How to ignore tmpfs, udf, iso9660 when dealing with filesystem metrics

How to avoid issues using Foreman when installing Windows OS?

31.10.2022

Using Prometheus + Alertmanager + Node Exporter to monitor a company's geo-distributed infrastructure.

How to implement an effective monitoring system to track equipment operation and quickly detect server failures.

17.10.2022

Migrating Virtual Servers to oVirt

Develop and apply a convenient scheme for transferring servers from the old oVirt Engine 4.2 infrastructure to the new oVirt Engine 4.5 infrastructure.

11.10.2022

Wazuh from the point of view of a Windows Administrator

How to choose a solution that will ensure a stable file configuration on servers and reliable alerts in event these files change.

09.10.2022

ELK-RabbitMQ Architecture - log management for a large IT infrastructure.

How to use the AMQP RabbitMQ broker to create a fault-tolerant architecture with minimal loss of log data during failures.

HOSTKEY Dedicated servers and cloud solutions Pre-configured and custom dedicated servers. AMD, Intel, GPU cards, Free DDoS protection amd 1Gbps unmetered port 30
4.3 67 67
Upload