Deployment Overview of Open WebUI and Ollama on Server¶

Prerequisites and Basic Requirements¶

The deployment requires a Linux server running Ubuntu 22.04 or Ubuntu 24.04. Ubuntu 25.04 is not supported due to the lack of official NVIDIA CUDA support for non-LTS releases. The system must have root privileges to install drivers, configure services, and manage containers.

The following hardware and software components are required: - NVIDIA GPU with compatible drivers for CUDA acceleration. - Docker Engine installed and running. - NVIDIA Container Toolkit or nvidia-docker2 package installed to enable GPU access within containers. - Network access to download CUDA repositories, Docker images, and model weights. - A valid domain name configured to point to the server's IP address for SSL certificate generation.

File and Directory Structure¶

The application utilizes specific directories for configuration, data storage, and certificates. The primary locations are:

• /root/nginx/: Contains the Docker Compose configuration for the proxy and SSL management.
• /data/nginx/user_conf.d/: Stores custom Nginx configuration files, including the host-specific proxy rules.
• /data/nginx/nginx-certbot.env: Environment file containing settings for the Nginx-Certbot container.
• /etc/systemd/system/ollama.service: Systemd unit file defining the Ollama service parameters.
• /app/backend/data: Persistent volume mount point inside the Open WebUI container for storing application data.
• /etc/letsencrypt: Volume mount point for storing SSL certificates managed by Certbot.

Application Installation Process¶

The installation involves setting up the NVIDIA CUDA environment, installing the Ollama service, and deploying the Open WebUI application via Docker.

CUDA and Driver Setup The system installs the CUDA toolkit and NVIDIA drivers based on the Ubuntu version. For Ubuntu 22.04, the HWE kernel is updated, and nvidia-docker2 is installed. For Ubuntu 24.04, the nvidia-container-toolkit is installed. The nouveau kernel module is removed to prevent conflicts with proprietary NVIDIA drivers.

Ollama Installation The Ollama service is installed using the official installation script. A dedicated system user named ollama is created. The default systemd service file is backed up and replaced with a custom configuration that sets the following environment variables: - OLLAMA_HOST=0.0.0.0 - OLLAMA_ORIGINS=* - LLAMA_FLASH_ATTENTION=1 - PATH includes /usr/local/cuda/bin for CUDA binary access.

After configuration, the ollama service is enabled and started. The default model qwen3:14b is pulled immediately after the service starts.

Open WebUI Deployment The Open WebUI application is deployed using Docker Compose. The configuration file is generated at /root/nginx/compose.yml. This file defines two services: - open-webui: Runs the ghcr.io/open-webui/open-webui:main image. - nginx: Runs the jonasal/nginx-certbot:latest image to handle reverse proxying and SSL.

Docker Containers and Their Deployment¶

The application relies on two Docker containers managed via a single docker compose file located at /root/nginx/compose.yml.

Open WebUI Container - Image: ghcr.io/open-webui/open-webui:main - Container Name: open-webui - Restart Policy: always - Volumes: Mounts the open-webui named volume to /app/backend/data for persistent data storage. - Extra Hosts: Configures host.docker.internal to resolve to the host gateway.

Nginx-Certbot Container - Image: jonasal/nginx-certbot:latest - Restart Policy: unless-stopped - Ports: Exposes ports 80 and 443 on the host. - Volumes: - nginx_secrets (external) mounted to /etc/letsencrypt. - /data/nginx/user_conf.d mounted to /etc/nginx/user_conf.d. - Environment: Uses an environment file at /data/nginx/nginx-certbot.env and sets [email protected]. - Dependencies: Starts after the open-webui service.

To start the containers, the command docker compose up -d is executed from the /root/nginx directory.

Proxy Servers¶

The deployment uses an Nginx container with integrated Certbot to manage SSL certificates and reverse proxy traffic.

• SSL Management: The jonasal/nginx-certbot image automatically handles Let's Encrypt certificate issuance and renewal.
• Proxy Configuration: A custom Nginx configuration file is generated at /data/nginx/user_conf.d/<prefix><server_id>.hostkey.in.conf.
• Routing: The configuration includes a location / block that proxies requests to the Open WebUI container at http://open-webui:8080.
• Ports: The server listens on standard HTTP (80) and HTTPS (443) ports.

Access Rights and Security¶

The system enforces specific user and file permissions to ensure secure operation.

• System Users: The ollama user is created as a system user to run the Ollama service.
• Directory Permissions:
• /root/nginx: Owned by root with mode 0755.
• /root/nginx/compose.yml: Owned by root with mode 0644.
• /etc/systemd/system/ollama.service: Owned by root with mode 0644.
• Firewall: The deployment exposes ports 80 and 443 for web traffic. Internal communication between containers occurs on port 8080 for Open WebUI.

Starting, Stopping, and Updating¶

Ollama Service Management The Ollama service is managed via systemd. - To restart the service: systemctl restart ollama - To enable the service on boot: systemctl enable ollama - To reload the daemon after configuration changes: systemctl daemon-reload

Docker Container Management The Open WebUI and Nginx containers are managed via Docker Compose in the /root/nginx directory. - To start or restart containers: docker compose up -d - To stop containers: docker compose down - To update the application, the docker compose up -d command is re-run after updating the image tags or configuration files.

Model Updates To add or update AI models, the ollama pull command is used. For example, to pull the default model: ollama pull qwen3:14b.