Spark User Interface Security

Introduction

This document includes the steps to secure the user interface for a Spark Standalone installation on the Gluent Node.

Authentication

Configure Spark Standalone

The following parameter in /opt/gluent/transport/spark/conf/spark-defaults.conf sets the username and password for the Spark User Interface:

Parameter

Value

spark.com.gluent.SparkBasicAuth.params

username=gluent,password=change_me

The values for gluent and change_me should be replaced with chosen values.

Restart Spark Standalone

For the changes to take effect Spark Standalone must be restarted.

To stop Spark Standalone issue the following commands:

$ $SPARK_HOME/sbin/stop-all.sh
$ $SPARK_HOME/sbin/stop-history-server.sh

To start Spark Standalone manually, issue the following commands:

$ $SPARK_HOME/sbin/start-all.sh
$ $SPARK_HOME/sbin/start-history-server.sh

A simple HTTP user/password will now be prompted for all Spark UIs.

SSL

The most basic steps to configure the key stores and the trust store for a Spark Standalone deployment mode is as follows:

  • Generate a key pair for each node

  • Export the public key of the key pair to a file on each node

  • Import all exported public keys into a single trust store

  • Distribute the trust store to the cluster nodes

Generate Certificate

Create a self-signed certificate as follows:

$ mkdir -p /opt/gluent/spark-cert
$ cd /opt/gluent/spark-cert

$ $JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -s) -storepass <password> -keystore spark-$(hostname -s).jks \
-keyalg RSA -keysize 2048 -validity 3650 \
-dname "cn=$(hostname -f)" -keypass <password>

$ $JAVA_HOME/bin/keytool -export -alias $(hostname -s) -storepass <password> -keystore spark-$(hostname -s).jks \
-file spark-$(hostname -s).cer

$ $JAVA_HOME/bin/keytool -importcert -alias $(hostname -s) -storepass <password> -keystore spark-$(hostname -s).truststore \
-file spark-$(hostname -s).cer -noprompt

The value for <password> should be replaced with a chosen value.

Configure Spark Standalone

Add the following parameters and values to /opt/gluent/transport/spark/conf/spark-defaults.conf:

Parameter

Value

spark.ssl.enabled

true

spark.ssl.trustStore

/opt/gluent/spark-cert/spark-<hostname>.truststore

spark.ssl.trustStorePassword

<password>

spark.ssl.keyStore

/opt/gluent/spark-cert/spark-<hostname>.jks

spark.ssl.keyStorePassword

<password>

spark.ssl.keyPassword

<password>

The value for <password> should be replaced with the chosen value when generating the certificate.

The value for <hostname> should be replaced with the result from hostname -s from the server.

Restart Spark Standalone

For the changes to take effect Spark Standalone must be restarted.

To stop Spark Standalone issue the following commands:

$ $SPARK_HOME/sbin/stop-all.sh
$ $SPARK_HOME/sbin/stop-history-server.sh

To start Spark Standalone manually, issue the following commands:

$ $SPARK_HOME/sbin/start-all.sh
$ $SPARK_HOME/sbin/start-history-server.sh

Spark UIs will now be accessible only via the HTTPS protocol.

Documentation Feedback

Send feedback on this documentation to: feedback@gluent.com