Skip to content

Deployment Overview of Apache Spark on Server

Prerequisites and Basic Requirements

To ensure a successful deployment, the following requirements must be met:

  • Operating System: Debian or Ubuntu.

  • Privileges: Root or sudo access is required for package installation and system configuration.

  • Java Runtime: The installation requires default-jdk.

  • Network/Ports:

  • Port 443 (HTTPS) must be open for external access via the proxy.

  • Internal ports used by Spark services include 8080, 8081, and 18080.

FQDN of the final panel on the hostkey.in domain

The application is accessible via a specific subdomain template based on the server ID.

Parameter Value
Prefix spark
Domain hostkey.in
Full template spark{Server_ID_from_Invapi}.hostkey.in

File and Directory Structure

The following directories are used for application files, configurations, and certificates:

  • /root/spark: A symbolic link to the active Apache Spark installation directory.

  • /root/nginx: Contains the Docker Compose configuration for the Nginx proxy.

  • /data/nginx/user_conf.d/: Stores the Nginx virtual host configuration files.

  • /etc/letsencrypt/live/: Location of SSL certificates (managed via Docker volume).

Application installation process

The installation follows a multi-step process involving system package management and manual binary extraction:

  1. System Update: The APT package manager is updated, and existing packages are upgraded to their latest versions.

  2. Java Installation: The default-jdk package is installed to provide the necessary Java environment.

  3. Environment Configuration:

  4. JAVA_HOME is set in /etc/environment (pointing to OpenJDK 11 or 17 depending on the OS version).

  5. SPARK_LOCAL_IP is set to 127.0.0.1 in /etc/environment.

  6. Binary Download: Apache Spark version 3.5.9 (bin-hadoop3) is downloaded from the official Apache archives directly to the /root directory.

  7. Extraction and Linking:

  8. The downloaded archive is extracted in /root.

  9. A symbolic link is created from /root/spark-3.5.9-bin-hadoop3 to /root/spark to provide a version-independent path for management.

  10. Cleanup: The downloaded .tgz archive is removed after successful extraction.

Access Rights and Security

  • Firewall: Only the external port (default 443) should be exposed to the public internet. All Spark service ports are handled internally via a reverse proxy.

  • User Permissions: Configuration files in /root/nginx and /data/nginx/user_conf.d/ are owned by root.

Databases

No external databases are required for this deployment as Apache Spark is running as a standalone application.

Docker Containers and Their Deployment

The deployment utilizes a single container to manage SSL certificates and reverse proxying:

  • Nginx Proxy Container

  • Image: jonasal/nginx-certbot:latest

  • Ports: Uses host network mode, listening on port 443.

  • Volumes:

  • nginx_secrets:/etc/letsencrypt (for SSL certificates)

  • /data/nginx/user_conf.d:/etc/nginx/user_conf.d (for site configurations)

  • Environment Variables: [email protected]

  • Restart Policy: unless-stopped

Custom Scripts and Additional Setup

The installation performs several environment setup actions:

  • System Reboot: A system reboot is performed after modifying /etc/environment to ensure the new JAVA_HOME and SPARK_LOCAL_IP settings are applied globally.

  • Nginx Configuration Generation: A custom Nginx configuration file is generated in /data/nginx/user_conf.d/ to map specific URL paths to Spark's internal services (Master, Worker, and History Server).

Application Update Instructions

To update the Apache Spark application:

  1. Download the new version of the Spark binary to the /root directory.

  2. Extract the archive.

  3. Update the symbolic link at /root/spark to point to the new version's directory.

Location of configuration files and data

  • Spark Installation: /root/spark

  • Nginx Compose File: /root/nginx/compose.yml

  • Nginx Site Configuration: /data/nginx/user_conf.d/spark{Server_ID_from_Invapi}.hostkey.in.conf

Available ports for connection

Service Internal Port External Path / URL
Web UI (Main) 4040 /
Master UI 8080 /master
Worker UI 8081 /worker
History Server 18080 /history

Starting and Stopping the application

The proxy service is managed via Docker Compose:

  • Start Proxy: Navigate to /root/nginx and run docker compose up -d.

  • Stop Proxy: Navigate to /root/nginx and run docker compose down.

question_mark
Is there anything I can help you with?
question_mark
AI Assistant ×