Deployment Overview of gpt-oss-20b on Server¶

Prerequisites and Basic Requirements¶

The deployment requires a server running the Ubuntu operating system. The following conditions must be met before proceeding:

• Root privileges or sudo access are required to install system packages, manage services, and configure Docker.
• Docker Engine must be installed and running on the server to host the application containers.
• The server must have access to the internet to download the Ollama installer, the gpt-oss:20b model, and Docker images.
• Port 8080 must be available for the Open WebUI application.
• Port 11434 is used internally by the Ollama service.
• Ports 80 and 443 are required for the Nginx proxy and SSL certificate management via Certbot.

File and Directory Structure¶

The application utilizes the following directory structure for configuration, data, and certificates:

• /root/nginx: Contains the Docker Compose configuration for the Nginx proxy and Certbot.
• /root/nginx/compose.yml: The Docker Compose file defining the Nginx service.
• /data/nginx/nginx-certbot.env: Environment file containing configuration for the Nginx-Certbot container.
• /data/nginx/user_conf.d: Directory storing custom Nginx configuration files for specific host keys.
• /etc/systemd/system/ollama.service: Systemd unit file for the Ollama service.
• /usr/share/ollama/.ollama/models: Default storage location for the downloaded gpt-oss:20b model.
• /var/lib/docker/volumes/open-webui: Docker volume storing persistent data for the Open WebUI application.

Application Installation Process¶

The deployment involves installing the Ollama inference engine, pulling the specific model, and launching the Open WebUI interface via Docker.

1. Install Ollama: The Ollama package is installed using the official installation script.

   curl -fsSL https://ollama.com/install.sh | sh
   

2. Create System User: A system user named ollama is created to run the service.
3. Configure Ollama Service: The ollama.service file is updated to expose the service on all network interfaces and enable specific environment variables:
   • OLLAMA_HOST=0.0.0.0
   • OLLAMA_ORIGINS=*
   • LLAMA_FLASH_ATTENTION=1
4. Pull Model: The gpt-oss:20b model is downloaded and stored in the default Ollama model directory.

   ollama pull gpt-oss:20b
   

5. Launch Open WebUI: The Open WebUI container is started with GPU acceleration enabled, connecting to the local Ollama instance.

Docker Containers and Their Deployment¶

Two primary Docker containers are deployed: one for the Open WebUI application and one for the Nginx proxy with Certbot.

Open WebUI Container¶

The Open WebUI container is launched using the following command parameters:

• Image: ghcr.io/open-webui/open-webui:cuda
• Container Name: open-webui
• Ports: Maps host port 8080 to container port 8080.
• GPU Access: The --gpus all flag is used to enable GPU acceleration.
• Host Resolution: The --add-host=host.docker.internal:host-gateway flag allows the container to reach the host machine.
• Volumes: A named volume open-webui is mounted to /app/backend/data for data persistence.
• Environment Variables:
  • ENV=dev
  • OLLAMA_BASE_URLS=http://host.docker.internal:11434
• Restart Policy: Set to always to ensure the container restarts automatically.

Nginx and Certbot Container¶

The Nginx proxy and SSL certificate management are handled by a container defined in a Docker Compose file located at /root/nginx/compose.yml.

• Image: jonasal/nginx-certbot:latest
• Restart Policy: unless-stopped
• Network Mode: host
• Volumes:
  • nginx_secrets (external) mounted to /etc/letsencrypt for SSL certificates.
  • /data/nginx/user_conf.d mounted to /etc/nginx/user_conf.d for custom configurations.
• Environment:
  • [email protected]
  • Configuration loaded from /data/nginx/nginx-certbot.env.

Proxy Servers¶

The Nginx proxy is configured to handle incoming traffic and manage SSL certificates using Certbot.

• Configuration Location: Custom Nginx configurations are stored in /data/nginx/user_conf.d.
• Proxy Pass: The proxy is configured to forward requests to the Open WebUI application running on the host.
  • Target: http://127.0.0.1:8080
• SSL Management: The nginx-certbot container automatically handles the generation and renewal of SSL certificates.
• Deployment: The proxy stack is started using the Docker Compose command within the /root/nginx directory.

  docker compose up -d
  

Access Rights and Security¶

Security and access control are managed through the following mechanisms:

• Ollama Service: The Ollama service runs under the dedicated ollama system user.
• Firewall: The deployment assumes that the firewall allows traffic on ports 80, 443, and 8080.
• Nginx Configuration: The Nginx configuration files in /data/nginx/user_conf.d are owned by root with permissions set to 0644.
• Directory Permissions: The /root/nginx directory is owned by root with permissions 0755.
• CORS: The Ollama service is configured with OLLAMA_ORIGINS=* to allow cross-origin requests from the web interface.

Starting, Stopping, and Updating¶

The services are managed using systemd for Ollama and docker compose for the Nginx proxy.

Ollama Service¶

• Restart Service:

  systemctl restart ollama
  

• Enable Service on Boot:

  systemctl enable ollama
  

• Reload Daemon: Required after modifying the service file.

  systemctl daemon-reload
  

Nginx Proxy¶

• Start/Update: Navigate to the configuration directory and run the compose command.

  cd /root/nginx
  docker compose up -d
  

• Stop:

  cd /root/nginx
  docker compose down
  

Open WebUI Container¶

• Start: The container is started via the docker run command. If the container is already running, the command will indicate that the port is in use.
• Stop:

  docker stop open-webui
  

• Remove:

  docker rm open-webui