Deployment Overview of gpt-oss-120b on Server¶

Prerequisites and Basic Requirements¶

To ensure a successful deployment, the following requirements must be met:

Operating System: Ubuntu (recommended).
Privileges: Root or sudo access is required for package installation and service management.
Hardware Acceleration: NVIDIA GPU support is required for CUDA-enabled container execution.
Network Ports:
443 (HTTPS) - External access via Nginx proxy.
8080 (HTTP) - Internal application traffic.

FQDN of the final panel on the hostkey.in domain¶

The application is accessible via a specific subdomain template based on the server ID.

Parameter	Value
Prefix	`gpt-oss`
Domain	`hostkey.in`
Full template	`gpt-oss{Server_ID_from_Invapi}.hostkey.in`

File and Directory Structure¶

The deployment utilizes the following directory structure for configuration and data persistence:

/root/nginx: Contains the Nginx Docker Compose configuration.
/data/nginx/user_conf.d: Stores custom Nginx configuration files, including proxy settings.
/data/nginx/nginx-certbot.env: Environment file for Certbot operations.
/var/lib/docker/volumes/open-webui/_data: Persistent data volume for the Open WebUI container.

Application installation process¶

The installation follows a multi-stage process involving system package management, driver configuration, and container orchestration:

System Preparation: The system performs dependency fixes via apt-get and purges conflicting NVIDIA 535 drivers to ensure compatibility with required CUDA versions.
Ollama Installation: The official Ollama installation script is executed. A dedicated ollama system user is created, and the ollama.service is modified to allow remote connections (OLLAMA_HOST=0.0.0.0) and enable Flash Attention (LLAMA_FLASH_ATTENTION=1).
Model Retrieval: The specific model gpt-oss:20b is pulled via the Ollama CLI.
Docker Deployment:
Docker and NVIDIA CUDA drivers are installed to support GPU passthrough.
The Open WebUI container is deployed using a docker run command with --gpus all enabled.
Proxy Configuration: An Nginx instance is configured via Docker Compose to act as a reverse proxy, routing traffic from the external domain to the internal application port.

Access Rights and Security¶

User Management: A system user named ollama is created for service execution.
Firewall/Network: The application listens on port 8080 internally and uses Nginx to expose the service securely over port 443.
Container Isolation: Open WebUI runs in a dedicated Docker container with restricted access to host networking via the host.docker.internal:host-gateway bridge.

Databases¶

This deployment does not utilize a standalone external database; instead, it relies on the internal storage mechanisms of the application containers and Ollama's local model storage.

Docker Containers and Their Deployment¶

The following containers are deployed as part of the application stack:

Open WebUI Container

Image: ghcr.io/open-webui/open-webui:cuda
Ports: 8080:8080
Volumes: open-webui:/app/backend/data
Environment Variables:
ENV='dev'
OLLAMA_BASE_URLS='http://host.docker.internal:11434'
Restart Policy: always

Nginx Proxy Container

Image: jonasal/nginx-certbot:latest
Network Mode: host
Volumes:
nginx_secrets:/etc/letsencrypt
/data/nginx/user_conf.d:/etc/nginx/user_conf.d
Restart Policy: unless-stopped

Custom Scripts and Additional Setup¶

The installation performs several non-standard configuration steps:

Service Modification: The ollama.service file is modified to include specific environment variables for network accessibility and performance optimization.
Proxy Injection: A configuration script modifies the Nginx user configuration files in /data/nginx/user_conf.d/ to inject a proxy_pass directive pointing to http://127.0.0.1:8080.

Application Update Instructions¶

To update the main application (Open WebUI), perform the following steps:

docker compose pull && docker compose up -d

(Note: This assumes the user is in the directory containing the relevant compose file or manages the container via Docker CLI).

Location of configuration files and data¶

Nginx Configuration: /data/nginx/user_conf.d/
Open WebUI Data: Managed via Docker volume open-webui.
Ollama Models: /usr/share/ollama/.ollama/models/

Available ports for connection¶

HTTPS (External): 443
HTTP (Internal): 8080

Starting and Stopping the application¶

To manage the Nginx proxy service:

cd /root/nginx
docker compose up -d # To start
docker compose down # To stop

The Open WebUI container is configured with a restart policy of always, meaning it will automatically start upon system boot or after a crash.