Deployment Overview of Apache Spark on Server¶
Prerequisites and Basic Requirements¶
To ensure a successful deployment, the following requirements must be met:
-
Operating System: Debian or Ubuntu.
-
Privileges: Root or sudo access is required for package installation and system configuration.
-
Java Runtime: The installation requires
default-jdk. -
Network/Ports:
-
Port
443(HTTPS) must be open for external access via the proxy. -
Internal ports used by Spark services include
8080,8081, and18080.
FQDN of the final panel on the hostkey.in domain¶
The application is accessible via a specific subdomain template based on the server ID.
| Parameter | Value |
|---|---|
| Prefix | spark |
| Domain | hostkey.in |
| Full template | spark{Server_ID_from_Invapi}.hostkey.in |
File and Directory Structure¶
The following directories are used for application files, configurations, and certificates:
-
/root/spark: A symbolic link to the active Apache Spark installation directory. -
/root/nginx: Contains the Docker Compose configuration for the Nginx proxy. -
/data/nginx/user_conf.d/: Stores the Nginx virtual host configuration files. -
/etc/letsencrypt/live/: Location of SSL certificates (managed via Docker volume).
Application installation process¶
The installation follows a multi-step process involving system package management and manual binary extraction:
-
System Update: The APT package manager is updated, and existing packages are upgraded to their latest versions.
-
Java Installation: The
default-jdkpackage is installed to provide the necessary Java environment. -
Environment Configuration:
-
JAVA_HOMEis set in/etc/environment(pointing to OpenJDK 11 or 17 depending on the OS version). -
SPARK_LOCAL_IPis set to127.0.0.1in/etc/environment. -
Binary Download: Apache Spark version
3.5.9(bin-hadoop3) is downloaded from the official Apache archives directly to the/rootdirectory. -
Extraction and Linking:
-
The downloaded archive is extracted in
/root. -
A symbolic link is created from
/root/spark-3.5.9-bin-hadoop3to/root/sparkto provide a version-independent path for management. -
Cleanup: The downloaded
.tgzarchive is removed after successful extraction.
Access Rights and Security¶
-
Firewall: Only the external port (default
443) should be exposed to the public internet. All Spark service ports are handled internally via a reverse proxy. -
User Permissions: Configuration files in
/root/nginxand/data/nginx/user_conf.d/are owned byroot.
Databases¶
No external databases are required for this deployment as Apache Spark is running as a standalone application.
Docker Containers and Their Deployment¶
The deployment utilizes a single container to manage SSL certificates and reverse proxying:
-
Nginx Proxy Container
-
Image:
jonasal/nginx-certbot:latest -
Ports: Uses
hostnetwork mode, listening on port443. -
Volumes:
-
nginx_secrets:/etc/letsencrypt(for SSL certificates) -
/data/nginx/user_conf.d:/etc/nginx/user_conf.d(for site configurations) -
Environment Variables:
[email protected] -
Restart Policy:
unless-stopped
Custom Scripts and Additional Setup¶
The installation performs several environment setup actions:
-
System Reboot: A system reboot is performed after modifying
/etc/environmentto ensure the newJAVA_HOMEandSPARK_LOCAL_IPsettings are applied globally. -
Nginx Configuration Generation: A custom Nginx configuration file is generated in
/data/nginx/user_conf.d/to map specific URL paths to Spark's internal services (Master, Worker, and History Server).
Application Update Instructions¶
To update the Apache Spark application:
-
Download the new version of the Spark binary to the
/rootdirectory. -
Extract the archive.
-
Update the symbolic link at
/root/sparkto point to the new version's directory.
Location of configuration files and data¶
-
Spark Installation:
/root/spark -
Nginx Compose File:
/root/nginx/compose.yml -
Nginx Site Configuration:
/data/nginx/user_conf.d/spark{Server_ID_from_Invapi}.hostkey.in.conf
Available ports for connection¶
| Service | Internal Port | External Path / URL |
|---|---|---|
| Web UI (Main) | 4040 | / |
| Master UI | 8080 | /master |
| Worker UI | 8081 | /worker |
| History Server | 18080 | /history |
Starting and Stopping the application¶
The proxy service is managed via Docker Compose:
-
Start Proxy: Navigate to
/root/nginxand rundocker compose up -d. -
Stop Proxy: Navigate to
/root/nginxand rundocker compose down.