Deployment Overview of Gemma4-26b on Server¶

Prerequisites and Basic Requirements¶

To ensure the successful deployment and operation of the application, the following requirements must be met:

Operating System: Ubuntu (recommended for driver compatibility).
Privileges: Root or sudo access is required for package installation and system configuration.
Hardware Requirement: NVIDIA GPU presence is highly recommended for hardware acceleration.
Ports:
443 (HTTPS): External access to the web interface.
8080: Internal application communication.
11434: Local Ollama service communication.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain template configured for the environment:

Parameter	Value
Prefix	`gemma26b`
Domain	`hostkey.in`
Full template	`gemma26b{Server_ID}.hostkey.in`

Application installation process¶

The installation follows a multi-stage process involving system preparation, driver configuration, and container orchestration:

System Preparation: The OS package cache is updated, and the system is upgraded to the latest distributed version. A Hardware Enablement (HWE) kernel is installed to ensure modern hardware support.
Driver Installation: If an NVIDIA GPU is detected via lspci, the recommended NVIDIA drivers are automatically installed using ubuntu-drivers autoinstall. The system undergoes a reboot to apply these changes.
Ollama Setup:
Ollama is installed via the official installation script (curl -fsSL https://ollama.com/install.sh | sh).
The service is enabled and started via systemd.
The specific LLM model gemma4:26b is pulled to the local storage.
Docker Environment: Docker is installed, and necessary Python modules for Docker management are added to the system.
Container Deployment: The OpenWebUI container is deployed via Docker to provide the user interface.

Access Rights and Security¶

Firewall/Ports: External traffic is routed through port 443 (HTTPS) managed by a reverse proxy.
Service Isolation: The application uses network_mode: host for the web interface container to facilitate communication with the local Ollama service on 127.0.0.1.

Docker Containers and Their Deployment¶

The deployment utilizes two primary containers to manage the user interface and SSL termination.

OpenWebUI Container¶

Image: ghcr.io/open-webui/open-webui:main
Container Name: open-webui
Ports: Uses host network mode (access via standard web ports).
Volumes:
open-webui:/app/backend/data (Persistent data storage)
Environment Variables:
OLLAMA_BASE_URL: http://127.0.0.1:11434
Restart Policy: always

Nginx Certbot Container¶

Image: jonasal/nginx-certbot:latest
Container Name: nginx
Ports: Uses host network mode (listens on port 80 and 443).
Volumes:
nginx_secrets:/etc/letsencrypt
/data/nginx/user_conf.d:/etc/nginx/user_conf.d
Restart Policy: unless-stopped
Environment Variables:
CERTBOT_EMAIL: [email protected]

Custom Scripts and Additional Setup¶

The following actions are performed during the setup to ensure correct application routing:

Configuration Generation: A compose.yml file is generated in /root/nginx/ to manage the reverse proxy.
Reverse Proxy Routing: The configuration files located in /data/nginx/user_conf.d/ are modified to include a proxy_pass directive pointing to http://127.0.0.1:8080, ensuring that incoming HTTPS traffic is correctly routed to the application backend.

Application Update Instructions¶

To update the main web interface, use the following command in the directory containing the container configuration:

docker compose pull && docker compose up -d

Location of configuration files and data¶

Nginx Configuration Directory: /root/nginx
User SSL/Proxy Configurations: /data/nginx/user_conf.d/
OpenWebUI Persistent Data: Docker volume named open-webui

Available ports for connection¶

HTTPS (External): 443
Ollama API (Internal): 11434
Application Backend (Internal): 8080

Proxy Servers¶

The deployment uses an Nginx container with Certbot integration to handle SSL/TLS encryption. It serves as a reverse proxy, terminating HTTPS connections and forwarding requests to the application running on port 8080.