Deployment Overview of Phi-4-14b on Server¶

Prerequisites and Basic Requirements¶

To ensure the application functions correctly, the following requirements must be met by the host environment:

• Operating System: Ubuntu (recommended).

• Privileges: Root or sudo access is required for package installation and service management.

• Hardware Acceleration: NVIDIA GPU support with CUDA installed is required for hardware acceleration.

• Network Ports:

• 443: External HTTPS traffic.

• 8080: Internal application traffic (Open WebUI).

• 11434: Ollama API communication.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain template based on your server ID:

Application installation process¶

The installation involves several stages to set up the environment, the AI engine, and the web interface:

1. System Preparation: Installation of Docker and NVIDIA CUDA drivers.

2. AI Engine Setup:

3. The Ollama service is installed via a shell script (curl -fsSL https://ollama.com/install.sh | sh).

4. A dedicated ollama system user is created.

5. The ollama.service configuration is modified to allow remote connections (OLLAMA_HOST=0.0.0.0) and enable Flash Attention (LLAMA_FLASH_ATTENTION=1).

6. The phi4 model is pulled via the command ollama pull phi4.

7. Web Interface Deployment: The Open WebUI application is deployed as a Docker container using the ghcr.io/open-webui/open-webui:cuda image.

Access Rights and Security¶

• Firewall: Ensure that port 443 is open to allow HTTPS traffic through the Nginx proxy.

• Service Isolation: The application uses a dedicated system user for Ollama and containerized isolation for the web interface.

• SSL/TLS: SSL certificates are managed via Certbot to secure the connection on the FQDN.

Databases¶

The application utilizes Docker volumes for data persistence rather than an external database server:

• open-webui: Used by the Open WebUI container to store application data and user information.

Docker Containers and Their Deployment¶

The deployment consists of two primary containers:

Nginx Proxy Container¶

• Image: jonasal/nginx-certbot:latest

• Network Mode: host

• Volumes:

• nginx_secrets:/etc/letsencrypt (SSL certificates)

• /data/nginx/user_conf.d:/etc/nginx/user_conf.d (Configuration files)

• Environment Variables:

• [email protected]

Open WebUI Container¶

• Image: ghcr.io/open-webui/open-webui:cuda

• Name: open-webui

• Ports: 8080:8080

• Volumes: open-webui:/app/backend/data

• Environment Variables:

• ENV='dev'

• OLLAMA_BASE_URLS='http://host.docker.internal:11434'

• Restart Policy: always

• Hardware Access: --gpus all

Custom Scripts and Additional Setup¶

The following configuration changes are applied during the setup process:

• Ollama Service Modification: The systemd service file for Ollama is updated to include specific environment variables (OLLAMA_HOST, OLLAMA_ORIGINS, and LLAMA_FLASH_ATTENTION) to facilitate communication with the Docker container.

• Nginx Proxy Configuration: A configuration file is generated in /data/nginx/user_conf.d/ that proxies incoming traffic from port 443 to the internal application running on http://127.0.0.1:8080.

Application Update Instructions¶

To update the main web interface, execute the following command in the directory containing your Docker configuration:

docker compose pull && docker compose up -d

Location of configuration files and data¶

• Nginx Configuration: /data/nginx/user_conf.d/

• Nginx Compose File: /root/nginx/compose.yml

• Ollama Service Config: /etc/systemd/system/ollama.service

• Open WebUI Data Volume: open-webui (Docker managed)

Available ports for connection¶

• HTTPS (External): 443

• Web Interface (Internal): 8080

Starting and Stopping the application¶

Open WebUI¶

To stop the web interface:

docker stop open-webui

docker start open-webui

Nginx Proxy¶

To manage the proxy service (located in /root/nginx):

cd /root/nginx
docker compose down
docker compose up -d