Deployment Overview of LiteLLM on Server¶

Prerequisites and Basic Requirements¶

To ensure a successful deployment, the server must meet the following requirements:

• Operating System: Ubuntu (the installation process is optimized for Ubuntu distributions).

• Privileges: Root or sudo access is required to install packages and manage Docker containers.

• Domain: A valid FQDN within the hostkey.in zone is required for SSL certificate provisioning via Certbot.

• Network Ports:

• Port 80/tcp: Required for HTTP traffic and Let's Encrypt challenges.

• Port 443/tcp: Required for HTTPS traffic.

• Port 4000/tcp: Internal application port used by the LiteLLM proxy and Admin UI.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a dynamically generated subdomain based on the server ID.

File and Directory Structure¶

The application uses the following directory structure for configuration, data persistence, and SSL certificates:

• /root/litellm: Main application directory containing environment variables (.env) and Docker Compose files.

• /root/nginx: Configuration directory for the Nginx reverse proxy.

• /data/nginx: User-defined Nginx configurations and Let's Encrypt challenge paths.

• /data/nginx/user_conf.d: Custom Nginx server block templates.

• /data/nginx/letsencrypt: Webroot for SSL certificate validation.

Application Installation Process¶

The installation is performed through a multi-step process that prepares the system environment and deploys the application stack:

1. System Preparation: The system updates package lists and installs essential dependencies, specifically ca-certificates and curl.

2. Docker Environment Setup: Any legacy Docker repository configurations are removed to prevent conflicts, and the official Docker engine is installed.

3. Application Directory Creation: A dedicated directory /root/litellm is created with restricted permissions (0750).

4. Environment Configuration:

5. A .env file is generated in /root/litellm.

6. This file contains a unique LITELLM_MASTER_KEY, LITELLM_SALT_KEY, and POSTGRES_PASSWORD.

7. It also defines the UI credentials (defaulting to admin).

8. Container Deployment: The application is deployed using Docker Compose, which pulls the required images and starts the services in the correct order.

Access Rights and Security¶

• Firewall Configuration: If UFW (Uncomplicated Firewall) is present on the system, rules are automatically added to allow incoming traffic on ports 80/tcp and 443/tcp.

• File Permissions: Sensitive files like .env are restricted with 0600 permissions.

• Network Isolation: The LiteLLM application container is configured to listen only on the loopback interface (127.0.0.1) for its management port, ensuring it is only accessible through the Nginx reverse proxy.

Databases¶

The application uses a PostgreSQL database for data persistence (storing models and credentials).

• Connection Method: The LiteLLM container connects to the database via the internal Docker network using the hostname db.

• Image: docker.io/library/postgres:16-alpine

• Database Name: litellm

• Storage Location: Persistent data is stored in a Docker volume named postgres-data.

Docker Containers and Their Deployment¶

The deployment consists of three primary containers managed via Docker Compose.

Application Stack (LiteLLM)¶

Reverse Proxy (Nginx)¶

Custom Scripts and Additional Setup¶

The deployment includes several automated configuration steps:

• Host File Modification: The final FQDN is added to the local /etc/hosts file pointing to the server's internal IP to facilitate local proxying.

• SSL Certificate Provisioning: A specialized Nginx container is used to automate the acquisition and renewal of SSL certificates via Certbot. It uses a 8-day renewal interval.

• Health Checks: The system performs automated checks to ensure the web panel is responding over both HTTP (for redirection) and HTTPS before completing the setup.

Application Update Instructions¶

To update the LiteLLM application, navigate to the application directory and pull the latest images:

cd /root/litellm
docker compose pull && docker compose up -d

Location of Configuration Files and Data¶

• Application Environment: /root/litellm/.env

• Nginx User Configs: /data/nginx/user_conf.d/

• PostgreSQL Data: Managed via Docker volume postgres-data.

• SSL Certificates: Managed via Docker volume nginx_secrets.

Available Ports for Connection¶

• HTTPS (Secure Access): Port 443 via the configured FQDN.

• HTTP (Redirect to HTTPS): Port 80.