Deployment Overview of gpt-oss on Server¶

Prerequisites and Basic Requirements¶

The server must meet the following requirements to support the deployment:

• Operating System: Ubuntu

• Privileges: Root access or sudo privileges are required for installation and configuration.

• Network Access: The server must have internet access to download Docker images, models, and certificates.

• Ports:

• Internal application port: 8080

• External HTTPS port: 443

• Ollama service port: 11434

FQDN of the Final Panel¶

The application is accessible via the fully qualified domain name on the hostkey.in domain using the following format: gpt-oss<Server ID>.hostkey.in:443

Replace <Server ID> with the specific server identifier assigned to the host.

File and Directory Structure¶

The deployment utilizes the following directory structure for configurations, data, and certificates:

• /root/nginx/: Contains the Docker Compose file for the reverse proxy and SSL management.

• /root/nginx/compose.yml: Docker Compose configuration for Nginx and Certbot.

• /data/nginx/: Stores Nginx data and user configurations.

• /data/nginx/nginx-certbot.env: Environment file for Certbot.

• /data/nginx/user_conf.d/: Directory containing specific server configuration files.

• /data/nginx/user_conf.d/gpt-oss<Server ID>.hostkey.in.conf: The Nginx configuration file for this specific application.

• /usr/share/ollama/.ollama/models/: Storage location for the gpt-oss model.

• /var/lib/docker/volumes/open-webui/_data: Docker volume for Open WebUI persistent data.

• /etc/systemd/system/ollama.service: Systemd unit file for the Ollama service.

• /etc/systemd/system/ollama.service.bak: Backup of the original Ollama service file.

Application Installation Process¶

The application consists of two main components: the backend AI engine (Ollama) and the frontend interface (Open WebUI).

1. Ollama Installation: The Ollama service is installed via a shell script from the official source.

   curl -fsSL https://ollama.com/install.sh | sh
   

2. Ollama Configuration: The ollama.service file is modified to include specific environment variables:

   • OLLAMA_HOST=0.0.0.0

   • OLLAMA_ORIGINS=*

   • LLAMA_FLASH_ATTENTION=1 The service is then reloaded and restarted to apply these changes.

3. Model Deployment: The gpt-oss:20b model is pulled and stored locally.

   ollama pull gpt-oss:20b
   

4. Frontend Deployment: The Open WebUI frontend is deployed as a Docker container named open-webui with the cuda tag, mapped to port 8080.

Access Rights and Security¶

Security and access are managed through the following mechanisms:

• User Creation: A system user named ollama is created to manage the Ollama service.

• Firewall: External traffic is routed through Nginx, which handles SSL termination. Direct access to internal ports is not exposed publicly.

• Container Restrictions: The Open WebUI container runs with the --gpus all flag to enable hardware acceleration and uses a host gateway for internal communication.

Databases¶

The Open WebUI application uses a local Docker volume for data persistence rather than a separate database server.

• Storage Location: /var/lib/docker/volumes/open-webui/_data

• Mount Point: The volume is mounted at /app/backend/data inside the container.

• Connection Method: Internal file-based storage managed by the application.

Docker Containers and Their Deployment¶

The deployment utilizes two primary containerized components:

1. Open WebUI: Deployed using a standalone docker run command.

   docker run -d -p 8080:8080 --gpus all \
     --add-host=host.docker.internal:host-gateway \
     -v open-webui:/app/backend/data \
     --name open-webui \
     -e ENV='dev' \
     -e OLLAMA_BASE_URLS='http://host.docker.internal:11434' \
     --restart always ghcr.io/open-webui/open-webui:cuda
   

2. Nginx and Certbot: Deployed using Docker Compose located at /root/nginx/compose.yml. This setup handles SSL certificates and reverse proxying.

   docker compose up -d
   

   This command is executed from the /root/nginx directory.

Proxy Servers¶

An Nginx reverse proxy with integrated Certbot is used to manage SSL certificates and forward traffic to the internal application.

• Service: Nginx (via jonasal/nginx-certbot:latest image).

• SSL: Managed automatically by Certbot using the domain gpt-oss<Server ID>.hostkey.in.

• Proxy Configuration:

• The Nginx configuration file gpt-oss<Server ID>.hostkey.in.conf is located in /data/nginx/user_conf.d/.

• The configuration includes a proxy_pass directive pointing to the internal application:

  proxy_pass http://127.0.0.1:8080;
  

• Domain Mapping: Traffic to gpt-oss<Server ID>.hostkey.in is forwarded to the internal port 8080.

Permission Settings¶

The following file and directory permissions are applied:

• /root/nginx/: 0755 (Owner: root, Group: root).

• /root/nginx/compose.yml: 0644 (Owner: root, Group: root).

• /etc/systemd/system/ollama.service: Managed by systemd with root ownership.

• Docker volumes and service data are managed under the root user and specific service users (e.g., ollama).

Location of Configuration Files and Data¶

Configuration files and data are located in the following paths:

Available Ports for Connection¶

The application exposes the following ports for access:

Starting, Stopping, and Updating¶

Management of the services is performed using standard system and Docker commands.

Ollama Service:

• Start: systemctl start ollama

• Stop: systemctl stop ollama

• Restart: systemctl restart ollama

• Enable on Boot: systemctl enable ollama

Open WebUI Container:

• Start: docker start open-webui

• Stop: docker stop open-webui

• Restart: docker restart open-webui

• Logs: docker logs open-webui

Nginx and Certbot (Docker Compose):

• Start: docker compose up -d (executed in /root/nginx)

• Stop: docker compose down

• Restart: docker compose restart

• Logs: docker compose logs -f