Deployment Overview of Apache Spark on Server¶

Prerequisites and Basic Requirements¶

To ensure a successful deployment, the server must meet the following requirements:

Operating System: Debian or Ubuntu.
Privileges: Root or sudo access is required for package installation and system configuration.
Java Runtime: Java (OpenJDK) must be installed. The specific version depends on the OS distribution:
For Ubuntu/Debian Bullseye: default-jdk (Java 11).
For Debian Bookworm: OpenJDK 17.
Network Requirements:
Access to archive.apache.org to download Spark binaries.
Outbound access for SSL certificate acquisition via Certbot.

The application is accessible via a specific subdomain template based on the server ID.

The deployment utilizes several specific directories for configuration, data, and proxy management:

/root/nginx: Contains the Nginx Docker Compose file.
/data/nginx/user_conf.d/: Stores the Nginx virtual host configuration files.
/etc/environment: Used to define system-wide environment variables such as JAVA_HOME and SPARK_LOCAL_IP.

The installation of Apache Spark 3.5.3 is performed through a series of automated steps:

System Update: The package manager cache is updated, and all existing packages are upgraded to their latest versions.
Java Installation: The default-jdk package is installed to provide the necessary runtime environment.
Environment Configuration:
JAVA_HOME is set in /etc/environment pointing to the appropriate OpenJDK path.
SPARK_LOCAL_IP is set to 127.0.0.1.
Binary Download and Extraction:
The Apache Spark 3.5.3 binary archive (spark-3.5.3-bin-hadoop3.tgz) is downloaded from the official Apache archives.
The archive is extracted to the system.
The downloaded .tgz file is removed after extraction to clean up the root directory.
System Reboot: A system reboot is performed to ensure all environment variable changes are applied globally.

Firewall/Ports: The application uses a reverse proxy to handle incoming traffic on port 443 (HTTPS).
SSL/TLS: SSL certificates are managed via Certbot within a Docker container to secure the connection.
Environment Isolation: System environment variables are configured to ensure Spark services bind correctly to local interfaces where necessary.

The deployment utilizes a Docker container to manage Nginx and SSL certificate renewal through a reverse proxy setup.

Container Name	Image	Ports/Network	Volumes	Environment Variables	Restart Policy
`nginx`	`jonasal/nginx-certbot:latest`	`host` mode	`- nginx_secrets:/etc/letsencrypt` `- /data/nginx/user_conf.d:/etc/nginx/user_conf.d`	`[email protected]`	`unless-stopped`

To update the Apache Spark application:

Since the application is installed via manual binary extraction and system package management, updates require downloading the new version of the Spark binaries and replacing the existing installation files.
For the Nginx proxy component, use the following command in the /root/nginx directory:
```
docker compose pull && docker compose up -d
```

Nginx Proxy Config: /data/nginx/user_conf.d/spark{Server_ID}.hostkey.in.conf
Docker Compose File: /root/nginx/compose.yml
SSL Certificates: Managed within the nginx_secrets Docker volume and accessible to Nginx at /etc/letsencrypt/live/.

The following ports are used by the application services (accessible via the proxy):

The application is served through an Nginx reverse proxy running in a Docker container. This proxy handles:

SSL termination using certificates obtained via Certbot.
Routing requests to the appropriate Spark sub-services (Master, Worker, History) based on the URL path.
WebSocket support for real-time communication.