Deployment Overview of Qwen3-32B on Server¶

Prerequisites and Basic Requirements¶

To ensure the successful deployment and operation of the application, the following requirements must be met:

• Operating System: Ubuntu (recommended).

• Privileges: Root or sudo access is required for package installation and service configuration.

• Hardware Acceleration: NVIDIA GPU with CUDA support is required for optimized model inference.

• Network Ports:

• 443 (HTTPS): External access to the web interface.

• 8080: Internal communication for the Web UI.

• 11434: Local Ollama API service.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain template based on your server ID.

File and Directory Structure¶

The deployment utilizes the following directory structure for configuration and data persistence:

• /root/nginx: Contains the Nginx Docker Compose configuration.

• /data/nginx/user_conf.d: Stores Nginx virtual host configuration files.

• /data/nginx/nginx-certbot.env: Configuration file for Certbot environment variables.

• /var/lib/docker/volumes/open-webui/_data: Persistent storage for the Open WebUI application data.

Application installation process¶

The installation follows a multi-step process involving system package management, service configuration, and container orchestration:

1. System Dependencies: The curl package is installed via the apt package manager to facilitate script execution.

2. Ollama Installation:

3. Ollama is installed using the official installation script via curl.

4. A dedicated system user named ollama is created.

5. The ollama.service configuration is modified to allow remote connections (OLLAMA_HOST=0.0.0.0) and enable Flash Attention (LLAMA_FLASH_ATTENTION=1).

6. Model Acquisition: The specific model qwen3:32b is pulled and downloaded via the Ollama CLI.

7. Docker Deployment:

8. A Docker container for Open WebUI is launched using a docker run command with GPU support enabled (--gpus all).

9. Proxy Configuration: An Nginx instance is deployed via Docker Compose to handle SSL termination and proxy requests to the application.

Access Rights and Security¶

• Firewall: Ensure that port 443 is open for external HTTPS traffic.

• Service Isolation: The Ollama service is configured to listen on all interfaces (0.0.0.0) to allow communication between the host and the Docker container via the host.docker.internal gateway.

• User Management: A system user ollama is created to manage the backend engine.

Databases¶

The application uses a Docker volume for data persistence:

• Volume Name: open-webui

• Purpose: Stores all application-specific data, including user accounts and chat history.

Docker Containers and Their Deployment¶

The deployment consists of two primary containers:

Open WebUI Container¶

• Image: ghcr.io/open-webui/open-webui:cuda

• Ports: 8080:8080

• Volumes: open-webui:/app/backend/data

• Environment Variables:

• ENV=dev

• OLLAMA_BASE_URLS=http://host.docker.internal:11434

• Restart Policy: always

• Additional Flags: --add-host=host.docker.internal:host-gateway, --gpus all

Nginx Proxy Container¶

• Image: jonasal/nginx-certbot:latest

• Network Mode: host

• Volumes:

• nginx_secrets:/etc/letsencrypt

• /data/nginx/user_conf.d:/etc/nginx/user_conf.d

• Restart Policy: unless-stopped

Custom Scripts and Additional Setup¶

The following actions are performed during the setup process:

• Ollama Service Optimization: The systemd service file for Ollama is updated to include specific environment variables required for high-performance inference.

• Nginx Proxy Mapping: A configuration update is applied to the Nginx user configuration files to route traffic from the web domain to http://127.0.0.1:8080.

Application Update Instructions¶

To update the main application (Open WebUI), execute the following command:

docker compose pull && docker compose up -d

Location of configuration files and data¶

• Web UI Data: /var/lib/docker/volumes/open-webui/_data

• Nginx Configs: /data/nginx/user_conf.d

• Ollama Service: /etc/systemd/system/ollama.service

Available ports for connection¶

• 443: Secure HTTPS access (External).

• 8080: Web interface internal port.

• 11434: Ollama API port.

Starting and Stopping the application¶

Open WebUI:

• Start: docker start open-webui

• Stop: docker stop open-webui

Nginx Proxy:

• Start: Navigate to /root/nginx and run docker compose up -d

• Stop: Navigate to /root/nginx and run docker compose down