Deployment Overview of LiteLLM on Server¶

Prerequisites and Basic Requirements¶

Before installation, ensure the server meets the following requirements:

• Operating System: Ubuntu (recommended).

• Privileges: Root or sudo access is required for package installation and Docker management.

• Ports:

• 80/tcp (HTTP) for Let's Encrypt validation and redirection.

• 443/tcp (HTTPS) for secure web traffic.

• 4000/tcp (Internal) for the LiteLLM API and Admin UI.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain generated based on the server ID.

File and Directory Structure¶

The deployment utilizes the following directory structure for configuration and data persistence:

• /root/litellm: Main application directory containing environment variables and Docker Compose files.

• /root/nginx: Configuration directory for the Nginx reverse proxy.

• /data/nginx: User-defined Nginx configurations and SSL challenge paths.

• /data/nginx/user_conf.d: Custom Nginx server block templates.

• /data/nginx/letsencrypt: Let's Encrypt webroot for certificate validation.

Application Installation Process¶

The installation process follows these steps:

1. System Preparation: The system is updated, and necessary dependencies (ca-certificates, curl) are installed via the package manager.

2. Docker Environment Setup: Any legacy Docker repository sources or old keyrings are removed to ensure a clean environment. Docker is then installed on the host.

3. LiteLLM Stack Deployment:

4. A dedicated directory /root/litellm is created with restricted permissions (0750).

5. An .env file is generated containing unique secrets: LITELLM_MASTER_KEY, LITELLM_SALT_KEY, and POSTGRES_PASSWORD.

6. A Docker Compose stack is initialized using the docker compose up -d command.

7. Reverse Proxy Configuration:

8. Nginx and Certbot are deployed via Docker to handle SSL termination.

9. The server's /etc/hosts file is updated to resolve the final FQDN to the local IP.

10. UFW (Uncomplicated Firewall) rules are configured to allow traffic on ports 80 and 443.

Access Rights and Security¶

• Firewall: UFW is configured to permit incoming TCP traffic on ports 80 and 443.

• Permissions: Sensitive files like .env are restricted with 0600 permissions, and the application directory uses 0750.

• Secrets: High-entropy keys for LiteLLM encryption and database authentication are generated during the initial setup.

Databases¶

The application uses a PostgreSQL database to store model information and credentials.

Docker Containers and Their Deployment¶

The deployment consists of three primary containers.

LiteLLM Stack (via /root/litellm/compose.yml)¶

Database Container

• Image: docker.io/library/postgres:16-alpine

• Container Name: litellm-db

• Environment Variables: POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD

• Volumes: postgres-data:/var/lib/postgresql/data

• Restart Policy: unless-stopped

LiteLLM Application Container

• Image: docker.litellm.ai/berriai/litellm:main-stable

• Container Name: litellm

• Environment Variables: LITELLM_MODE=PRODUCTION, STORE_MODEL_IN_DB=True, DATABASE_URL, PORT=4000

• Ports: 127.0.0.1:4000 -> 4000 (Bound to loopback for security)

• Restart Policy: unless-stopped

Reverse Proxy Stack (via /root/nginx/compose.yml)¶

Nginx Certbot Container

• Image: jonasal/nginx-certbot:6.2.0-nginx1.31.0

• Network Mode: host

• Volumes:

• nginx_secrets:/etc/letsencrypt

• /data/nginx/user_conf.d:/etc/nginx/user_conf.d

• /data/nginx/letsencrypt:/var/www/letsencrypt

• Restart Policy: unless-stopped

Application Update Instructions¶

To update the LiteLLM application, navigate to the application directory and pull the latest images:

cd /root/litellm
docker compose pull && docker compose up -d