Deployment Overview of Self-hosted AI Chatbot (Ollama and Open WebUI) on Server¶

Prerequisites and Basic Requirements¶

The deployment requires a server running a supported version of Ubuntu. The system must have root privileges to execute installation scripts and manage services. The following components are mandatory:

Operating System: Ubuntu 22.04 or Ubuntu 24.04. Ubuntu 25.04 is not supported due to lack of official NVIDIA CUDA support for non-LTS releases.
Privileges: Root or sudo access is required for installing CUDA, Docker, and configuring system services.
Hardware: NVIDIA GPU with compatible drivers is required for running models efficiently.
Ports: Ports 80 and 443 must be open for the Nginx proxy and SSL certificate management. Port 8080 is used internally by the application.

FQDN of the Final Panel¶

The application is accessible via the following Fully Qualified Domain Name (FQDN) format:

ollama<Server ID>.hostkey.in

The service operates over HTTPS on port 443.

File and Directory Structure¶

The deployment utilizes specific directories for configuration, data storage, and certificates. The structure is as follows:

/root/nginx: Contains the Docker Compose configuration for the proxy and web interface.
/data/nginx/user_conf.d: Stores custom Nginx configuration files for the application.
/etc/letsencrypt: Holds SSL certificates managed by the Nginx container.
/app/backend/data: Persistent volume location for the Open WebUI application data inside the container.
/etc/systemd/system: Contains the service unit file for Ollama (ollama.service).

Application Installation Process¶

The installation involves setting up CUDA drivers, Docker, Ollama, and the web interface via Docker containers.

OS Compatibility Check: The installation script verifies the Ubuntu version. If Ubuntu 25.04 is detected, the installation aborts.
CUDA and NVIDIA Drivers:
The script installs gcc as a prerequisite.
CUDA Toolkit and drivers are installed based on the Ubuntu release version (22.04 or 24.04).
The nvidia-docker2 package (for Ubuntu 22.04) or nvidia-container-toolkit (for Ubuntu 24.04) is installed to enable GPU passthrough for Docker containers.
Ollama Installation:
Ollama is installed using the official script from https://ollama.com/install.sh.
A system user named ollama is created.
The default service file is backed up and replaced with a custom configuration to enable GPU acceleration and specific environment variables.
Model Initialization:
Upon successful installation, the script automatically pulls the qwen3:14b model.

Access Rights and Security¶

Security is enforced through firewall rules, user permissions, and service isolation.

System Users: A dedicated system user ollama is created to run the Ollama service, preventing execution as root.
SSL/TLS: SSL certificates are managed automatically by the nginx-certbot container, ensuring encrypted communication over HTTPS.
Network Isolation: The application containers communicate internally via Docker networks, exposing only the Nginx proxy to the public internet.

Databases¶

The Open WebUI application stores its data within the Docker volume structure.

Storage Location: Data is persisted in the open-webui volume, mapped to /app/backend/data inside the container.
Connection Method: The application uses an internal SQLite database stored in the application data directory. No external database configuration is required.

Docker Containers and Their Deployment¶

The application relies on two primary containers managed via Docker Compose:

Open WebUI:
Image: ghcr.io/open-webui/open-webui:main
Container Name: open-webui
Restart Policy: always
Volume Mount: open-webui:/app/backend/data
Internal Port: 8080
Nginx Proxy:
Image: jonasal/nginx-certbot:latest
Restart Policy: unless-stopped
Port Mapping: 80 and 443 exposed to the host.
Volume Mounts:
- nginx_secrets mapped to /etc/letsencrypt
- Host directory /data/nginx/user_conf.d mapped to /etc/nginx/user_conf.d
Deployment Command: The containers are started using the following command in the /root/nginx directory:
```
docker compose up -d
```

Proxy Servers¶

Nginx serves as the reverse proxy and SSL terminator for the application.

Configuration Location: Custom configurations are stored in /data/nginx/user_conf.d/ollama<Server ID>.hostkey.in.conf.
Proxy Settings:
The proxy forwards requests from location / to http://open-webui:8080.
SSL termination is handled by the Nginx container using Let's Encrypt.
Environment:
The email address for certificate management is set to [email protected].

Permission Settings¶

File and directory permissions are configured as follows:

Nginx Directory: /root/nginx is set to 0755 with ownership by root:root.
Compose File: /root/nginx/compose.yml is set to 0644 with ownership by root:root.
System Service: The Ollama service file at /etc/systemd/system/ollama.service is set to 0644 with ownership by root:root.

Location of Configuration Files and Data¶

Key configuration and data files are located at the following paths:

Docker Compose: /root/nginx/compose.yml
Nginx Custom Config: /data/nginx/user_conf.d/ollama<Server ID>.hostkey.in.conf
Ollama Service File: /etc/systemd/system/ollama.service
Application Data: /app/backend/data (inside the open-webui container volume)

Available Ports for Connection¶

The following ports are configured for the deployment:

Port 443: HTTPS traffic for the web interface (External).
Port 80: HTTP traffic for SSL certificate validation and redirection (External).
Port 3000: Internal port defined in configuration (not directly exposed to the public).
Port 8080: Internal port used by the Open WebUI container.

Starting, Stopping, and Updating¶

The Ollama service and Docker containers are managed via standard system and container commands.

Ollama Service Management: To manage the Ollama backend service, use the following systemctl commands:

systemctl restart ollama
systemctl status ollama

Docker Container Management: To manage the Open WebUI and Nginx containers, navigate to the /root/nginx directory and use docker compose:

Start/Restart Containers:
```
cd /root/nginx
docker compose up -d
```
Stop Containers:
```
docker compose down
```
Update Containers: Pull the latest images and restart:
```
docker compose pull
docker compose up -d
```