Deployment Overview of Apache Airflow on Server¶

Prerequisites and Basic Requirements¶

The following system requirements must be met for a successful deployment:

Operating System: Debian (Bookworm) or Ubuntu.
Privileges: Root or sudo access is required for package installation and service management.
Python Version: Python 3.10.
Database: PostgreSQL must be installed and running on the host.
Network Ports:
8080: Internal webserver port.
443: External HTTPS access via Nginx proxy.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain template based on the server ID:

Parameter	Value
Prefix	`airflow`
Domain	`hostkey.in`
Full template	`airflow{Server_ID_from_Invapi}.hostkey.in`

Application installation process¶

The application is installed using a combination of system package management and Python's specialized installation tools depending on the operating system version:

System Dependency Installation: The system updates the APT cache and installs essential libraries including libssl-dev, python3-pip, postgresql, curl, and sqlite3.
Airflow Core Installation:
On Ubuntu/Debian (Legacy): Apache Airflow is installed via pip using specific constraints for version 2.10.1 to ensure dependency stability.
On Debian Bookworm: The application is installed using pipx in a dedicated environment (/opt/pipx) with the binary located in /usr/local/bin.
Database Initialization: The system executes the migration command to prepare the PostgreSQL database schema.
User Configuration: An administrative user named admin is created within the Airflow ecosystem.
Service Configuration: Systemd service files are created for both the Webserver and the Scheduler to ensure they start automatically on boot.

Docker Containers and Their Deployment¶

The deployment utilizes a Docker container to manage SSL certificates and Nginx reverse proxying:

nginx-certbot
Image: jonasal/nginx-certbot:latest
Ports: Uses host networking (network_mode: host) to handle traffic on port 443.
Volumes:
nginx_secrets:/etc/letsencrypt: For SSL certificate storage.
/data/nginx/user_conf.d:/etc/nginx/user_conf.d: For custom Nginx configuration files.
Environment Variables:
[email protected]
Restart Policy: unless-stopped

Custom Scripts and Additional Setup¶

The deployment performs several automated setup actions:

Environment Path Configuration: For Debian Bookworm, the installation script ensures that the pipx binary directory is added to the system path.
Systemd Service Creation:
/etc/systemd/system/airflow-webserver.service: Manages the web interface on port 8080.
/etc/systemd/system/airflow-scheduler.service: Manages the background task scheduler.
Nginx Configuration Setup: A configuration directory is initialized at /root/nginx to manage the proxy and SSL certificates via Docker Compose.

Application Update Instructions¶

To update the main Apache Airflow application, follow these steps:

Re-run the installation command or script used during the initial deployment to pull the latest compatible versions of the core components.
For the Nginx/Proxy layer, navigate to /root/nginx and execute:
```
docker compose pull && docker compose up -d
```

Location of configuration files and data¶

Nginx Configuration: /data/nginx/user_conf.d
Docker Compose (Proxy): /root/nginx/compose.yml
SSL Certificates: Managed via the nginx_secrets Docker volume.

Available ports for connection¶

Port	Service	Access Type
`443`	Nginx Proxy (HTTPS)	External
`8080`	Airflow Webserver	Internal/Local

Starting and Stopping the application¶

The application is managed via systemd. Use the following commands to manage the core services:

Starting Services:

sudo systemctl start airflow-webserver
sudo systemctl start airflow-scheduler

Stopping Services:

sudo systemctl stop airflow-webserver
sudo systemctl stop airflow-scheduler

Checking Status:

sudo systemctl status airflow-webserver
sudo systemctl status airflow-scheduler