Deployment Overview of Prometheus on Server¶

Prerequisites and Basic Requirements¶

The deployment requires a Linux server running Ubuntu with root privileges. The system must have Docker installed and configured to manage containerized applications. The following network ports must be available: - Port 9090 for the Prometheus web interface. - Ports 80 and 443 for the Nginx reverse proxy and SSL termination.

File and Directory Structure¶

The application utilizes specific directories for configuration, data storage, and proxy management. The structure is organized as follows: - /root/nginx: Contains the Docker Compose configuration for the Nginx proxy and Certbot. - /root/nginx/compose.yml: The Docker Compose file defining the Nginx service. - /data/nginx/user_conf.d: Directory containing custom Nginx configuration files for specific host keys. - /data/nginx/nginx-certbot.env: Environment file for Nginx and Certbot settings. - /etc/prometheus: Mount point for the Prometheus configuration file. - /prometheus: Mount point for Prometheus time-series database storage.

Application Installation Process¶

The Prometheus application is deployed as a Docker container using the official image prom/prometheus. The installation process involves creating the necessary directories and generating the configuration file before starting the container.

The Prometheus configuration file is located at /etc/prometheus/prometheus.yml (mapped from the host). It includes the following default settings: - scrape_interval: Set to 15s. - evaluation_interval: Set to 15s. - job_name: Configured as prometheus targeting localhost:9090.

The container is started with the following arguments: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus - --web.console.libraries=/etc/prometheus/console_libraries - --web.console.templates=/etc/prometheus/consoles - --web.enable-lifecycle

Docker Containers and Their Deployment¶

Two primary Docker containers are deployed: Prometheus and Nginx with Certbot.

Prometheus Container¶

The Prometheus container is managed directly via Docker commands with the following specifications: - Image: prom/prometheus (version specified in deployment variables). - Name: prometheus. - Restart Policy: always. - Ports: Host port 9090 mapped to container port 9090. - Volumes: - Configuration file mounted to /etc/prometheus/prometheus.yml. - Data directory mounted to /prometheus.

Nginx and Certbot Container¶

The reverse proxy and SSL certificate management are handled by a Docker Compose stack located in /root/nginx. The stack uses the image jonasal/nginx-certbot:latest.

The compose.yml file defines the following service: - Service Name: nginx. - Image: jonasal/nginx-certbot:latest. - Restart Policy: unless-stopped. - Network Mode: host. - Environment: - CERTBOT_EMAIL: Set to [email protected]. - Loads environment variables from /data/nginx/nginx-certbot.env. - Volumes: - nginx_secrets (external volume) mounted to /etc/letsencrypt. - /data/nginx/user_conf.d mounted to /etc/nginx/user_conf.d.

To start the Nginx stack, the command docker compose up -d is executed from the /root/nginx directory.

Proxy Servers¶

Nginx acts as the reverse proxy for the Prometheus application, handling SSL termination via Certbot. The proxy configuration is dynamically updated to route traffic to the Prometheus container.

The proxy rule is configured in the file /data/nginx/user_conf.d/<prefix><server_id>.hostkey.in.conf. The configuration includes: - A location / block. - A proxy_pass directive pointing to http://127.0.0.1:9090.

This setup ensures that external requests to the domain are securely forwarded to the internal Prometheus instance running on port 9090.

Permission Settings¶

File and directory permissions are strictly defined to ensure secure operation: - The /root/nginx directory is owned by root:root with mode 0755. - The compose.yml file is owned by root:root with mode 0644. - The Prometheus data directory is owned by user ID 65534 and group ID 65534 with mode 0775. - Recursive permissions are applied to the Prometheus data directory to ensure all subdirectories and files inherit the correct ownership.

Starting, Stopping, and Updating¶

The services are managed using Docker and Docker Compose commands.

Prometheus¶

• Start: The container is started automatically with the docker_container module ensuring the state is started.
• Restart: The container is configured with a restart_policy of always, ensuring it restarts automatically on failure or system reboot.

Nginx Proxy¶

• Start: Execute docker compose up -d from the /root/nginx directory.
• Stop: Execute docker compose down from the /root/nginx directory.
• Update: To update the Nginx image or configuration, modify the compose.yml file or the environment variables, then run docker compose up -d.