Deployment Overview of Gemma4-31b on Server¶

Prerequisites and Basic Requirements¶

To ensure the application functions correctly, the following system requirements must be met:

• Operating System: Ubuntu (22.04 HWE Kernel recommended).

• Hardware Requirement: NVIDIA GPU is required for hardware acceleration.

• Privileges: Root or sudo access is required for package installation and service management.

• Network/Ports:

• Port 80: Used by the proxy for SSL certificate challenges.

• Port 443: External HTTPS access.

• Port 11434: Internal Ollama API communication.

• Port 8080: Internal application traffic.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain template:

Application installation process¶

The installation follows a multi-stage process involving system optimization, driver configuration, and container orchestration:

1. System Preparation: The OS package cache is updated, and the system is upgraded to the latest distributed version. The linux-generic-hwe-22.04 kernel is installed to ensure hardware compatibility.

2. Driver Installation: The system identifies NVIDIA hardware using lspci. If a GPU is detected, the recommended drivers are automatically installed via ubuntu-drivers autoinstall. A system reboot is performed to finalize driver integration.

3. Ollama Engine Setup:

4. Ollama is installed via the official installation script: curl -fsSL https://ollama.com/install.sh | sh.

5. The ollama service is enabled and started via systemd.

6. The LLM model gemma4:31b is automatically pulled to the local storage.

7. Container Deployment: Docker is installed, and the OpenWebUI container is deployed to provide the user interface.

Access Rights and Security¶

• Firewall/Network: The application uses host networking mode for the web interface to facilitate communication with the local Ollama service on 127.0.0.1:11434.

• SSL/TLS: SSL certificates are managed via a dedicated Nginx container using Certbot, ensuring all external traffic is encrypted over port 443.

Docker Containers and Their Deployment¶

The deployment utilizes two primary containers to manage the web interface and secure proxying:

OpenWebUI Container¶

• Image: ghcr.io/open-webui/open-webui:main

• Container Name: open-webui

• Network Mode: host

• Environment Variables:

• OLLAMA_BASE_URL: http://127.0.0.1:11434

• Volumes: open-webui:/app/backend/data

• Restart Policy: always

Nginx Proxy Container¶

• Image: jonasal/nginx-certbot:latest

• Container Name: nginx

• Network Mode: host

• Environment Variables:

• CERTBOT_EMAIL: [email protected]

• Volumes:

• nginx_secrets:/etc/letsencrypt

• /data/nginx/user_conf.d:/etc/nginx/user_conf.d

• Restart Policy: unless-stopped

Custom Scripts and Additional Setup¶

The following actions are performed during the initial setup to configure the environment:

• Nginx Configuration Injection: A configuration file is generated in /data/nginx/user_conf.d/. The deployment automatically updates this file to include a proxy_pass directive pointing to http://127.0.0.1:8080, routing external traffic to the application backend.

• Ollama Model Initialization: An automated command ollama pull gemma4:31b is executed to ensure the specific model weights are available immediately upon startup.

Application Update Instructions¶

To update the main user interface, navigate to the Nginx configuration directory and perform a compose update:

cd /root/nginx && docker compose pull && docker compose up -d

Location of configuration files and data¶

• Nginx Configuration Directory: /data/nginx/user_conf.d/

• Nginx Compose File: /root/nginx/compose.yml

• OpenWebUI Data Volume: open-webui (Docker managed volume)

• SSL Certificates: Managed via the nginx_secrets Docker volume.

Available ports for connection¶

Starting and Stopping the application¶

The web interface is managed via Docker:

• Start OpenWebUI: docker start open-webui

• Stop OpenWebUI: docker stop open-webui

• Restart Nginx Proxy: docker exec nginx-nginx-1 nginx -s reload