Deployment Overview of gpt-oss-20b on Server¶

Prerequisites and Basic Requirements¶

The deployment requires a Linux server running Ubuntu. The following conditions must be met before installation: - Root privileges or sudo access are required for all installation and configuration steps. - The curl utility must be installed on the system. - Docker and Docker Compose must be available on the host. - The server must have access to the internet to download models and container images. - The following ports must be available: - Port 8080 for the Open WebUI application. - Port 11434 for the Ollama service. - Ports 80 and 443 for the Nginx proxy and SSL certificates.

File and Directory Structure¶

The application utilizes the following directory structure for configuration, data, and certificates: - /root/nginx: Contains the Nginx proxy configuration and Docker Compose files. - /root/nginx/compose.yml: The Docker Compose definition for the Nginx and Certbot services. - /data/nginx/user_conf.d: Stores custom Nginx configuration files, including the host-specific configuration for the application. - /data/nginx/nginx-certbot.env: Environment file containing settings for the Nginx-Certbot container. - /etc/systemd/system/ollama.service: Systemd service file for the Ollama backend. - /usr/share/ollama/.ollama/models: Default storage location for the downloaded gpt-oss:20b model. - /var/lib/docker/volumes/open-webui/_data: Persistent volume for Open WebUI backend data.

Application Installation Process¶

The deployment involves installing the Ollama backend, pulling the specific AI model, and running the Open WebUI frontend via Docker.

Install Ollama: The Ollama service is installed using the official installation script:
```
curl -fsSL https://ollama.com/install.sh | sh
```
Configure Ollama Service: The ollama.service file is updated to allow external connections and enable specific optimizations. The following environment variables are added:
OLLAMA_HOST=0.0.0.0
OLLAMA_ORIGINS=*
LLAMA_FLASH_ATTENTION=1
Pull the Model: The gpt-oss:20b model is downloaded and stored locally:
```
ollama pull gpt-oss:20b
```

Deploy Open WebUI: The Open WebUI container is launched with CUDA support and connected to the local Ollama instance:

docker run -d -p 8080:8080 --gpus all \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  -e ENV='dev' \
  -e OLLAMA_BASE_URLS='http://host.docker.internal:11434' \
  --restart always ghcr.io/open-webui/open-webui:cuda

Docker Containers and Their Deployment¶

Two primary Docker components are deployed: the Open WebUI application and the Nginx proxy with Certbot.

Open WebUI Container:
Image: ghcr.io/open-webui/open-webui:cuda
Name: open-webui
Port Mapping: Host port 8080 maps to container port 8080.
GPU Access: Enabled via --gpus all.
Volume: A named volume open-webui is mounted to /app/backend/data for persistence.
Restart Policy: Set to always.
Nginx and Certbot Container:
Image: jonasal/nginx-certbot:latest
Deployment Method: Managed via docker compose in the /root/nginx directory.
Network Mode: host
Volumes:
- nginx_secrets (external) mounted to /etc/letsencrypt.
- /data/nginx/user_conf.d mounted to /etc/nginx/user_conf.d.
Environment: Uses [email protected] and loads variables from /data/nginx/nginx-certbot.env.

Proxy Servers¶

The Nginx proxy is configured to handle SSL termination and route traffic to the Open WebUI application.

Configuration Location: The proxy configuration is defined in /root/nginx/compose.yml and custom host rules are stored in /data/nginx/user_conf.d.
Proxy Pass: The Nginx configuration includes a location / block that forwards requests to the local Open WebUI instance:
```
proxy_pass http://127.0.0.1:8080;
```
SSL Management: The jonasal/nginx-certbot container automatically manages SSL certificates using Let's Encrypt. Certificates are stored in the nginx_secrets volume.
Email Configuration: Certificate notifications are sent to [email protected].

Starting, Stopping, and Updating¶

The services are managed using systemd for Ollama and docker compose for the Nginx proxy.

Ollama Service:
Reload the systemd daemon after configuration changes:
```
systemctl daemon-reload
```
Restart the service:
```
systemctl restart ollama
```
Enable the service to start on boot:
```
systemctl enable ollama
```
Nginx Proxy:
Start or restart the proxy stack from the configuration directory:
```
cd /root/nginx
docker compose up -d
```
Open WebUI Container:
The container is configured with --restart always, ensuring it restarts automatically if it stops or the server reboots.
To manually stop the container:
```
docker stop open-webui
```
To manually start the container:
```
docker start open-webui
```