Sqoop Authentication

Introduction

By default the ORA_APP_PASS password will be exposed in clear-text on the Sqoop command line. If the password has been encrypted in offload.env using Password Tool then it will be decrypted before being included on the Sqoop command line.

To prevent a clear password from being exposed on the Sqoop command line one of the following approaches can be used:

Sqoop Password File

Using a Sqoop password file removes the clear-text password from the Sqoop command line. It is important to note that password remains in clear-text in a file residing in HDFS.

The steps required to use a Sqoop password file with Offload are detailed below.

Create a Password File

On a node on which hdfs commands can be issued, create a password file and copy it to HDFS as the Gluent Data Platform software owner:

$ echo -n <ora_app_user_password> > ./gluent_app_credential.txt
$ hdfs dfs -put ./gluent_app_credential.txt /user/gluent/
$ hdfs dfs -chmod 400 /user/gluent/gluent_app_credential.txt

The value for <ora_app_user_password> should be replaced with the actual ORA_APP_PASS password.

Update Gluent Environment File

Set SQOOP_PASSWORD_FILE to /user/gluent/gluent_app_credential.txt in offload.env.

Important

Any changes made to the Gluent Data Platform environment file (offload.env) must be propagated across all installations.

Offload will now pass the file name to Sqoop and no clear-text passwords will be used.

Hadoop Credential Provider API

Using the Hadoop credential provider API removes the clear-text password from the Sqoop command line and stores the password in an encrypted key store.

For simplicity, the steps below assume the credential provider key store is not password protected. Refer to the note at the end of this section for further information on this topic.

Create a Password Alias

On a node on which hadoop commands can be issued, create a password alias in a key store as the Gluent Data Platform software owner:.

When prompted for the alias password enter the actual ORA_APP_PASS password:

$ hadoop credential create gluent.gluent_app -provider jceks://hdfs/user/gluent/gluent_app.jceks
WARNING: You have accepted the use of the default provider password
by not configuring a password in one of the two following locations:
    * In the environment variable HADOOP_CREDSTORE_PASSWORD
    * In a file referred to by the configuration entry
      hadoop.security.credstore.java-keystore-provider.password-file.
Please review the documentation regarding provider passwords in
the keystore passwords section of the Credential Provider API
Continuing with the default provider password.
Enter alias password:
Enter alias password again:
gluent.gluent_app has been successfully created.
Provider jceks://hdfs/user/gluent/gluent_app.jceks has been updated.

Check Hadoop documentation for more detail on the credential provider API. The key store is created with restricted permissions, so ensure that these are appropriate for your environment. Operating system permissions are the primary protection for credential provider key stores.

Update Gluent Environment File

Set OFFLOAD_TRANSPORT_PASSWORD_ALIAS to gluent.gluent_app in offload.env.

Sqoop also needs the location of the key store. If this is not already defined in the Hadoop core site property hadoop.security.credential.provider.path then it should be passed to Sqoop using OFFLOAD_TRANSPORT_CREDENTIAL_PROVIDER_PATH:

Set OFFLOAD_TRANSPORT_CREDENTIAL_PROVIDER_PATH to "jceks://hdfs/user/gluent/gluent_app.jceks" in offload.env.

Important

Any changes made to the Gluent Data Platform environment file (offload.env) must be propagated across all installations.

Offload will now pass the alias name to Sqoop and no clear-text passwords will be used.

Note

Using a Password Protected Key Store

If a key store is protected by a password then the password must be made available to Sqoop via one of the two methods below:

  1. Via the environment variable HADOOP_CREDSTORE_PASSWORD

  2. In a file referred to by the configuration entry hadoop.security.credstore.java-keystore-provider.password-file

This configuration is outside of Gluent Data Platform but the information below may be useful:

  • When using the hadoop.security.credstore.java-keystore-provider.password-file configuration entry Offload must be run from within the Hadoop cluster, for example on an edge node

  • When using the environment variable HADOOP_CREDSTORE_PASSWORD then Offload should either run in non-SSH mode or SSH must be configured to allow the environment variable to be forwarded using the SendEnv and AcceptEnv SSH directives

Oracle Wallet

Using an Oracle Wallet removes the clear-text password from the Sqoop command line and stores the password in an encrypted key store.

Create Oracle Wallet

On the Oracle Database server create the Oracle Wallet and add the Gluent Data Platform credential:

$ . $OFFLOAD_HOME/conf/offload.env
$ mkdir -p $OFFLOAD_HOME/../gluent_wallet
$ orapki wallet create -wallet $OFFLOAD_HOME/../gluent_wallet -auto_login -pwd <password>
$ mkstore -wrl "$OFFLOAD_HOME/../gluent_wallet" -createCredential "$ORA_CONN" "$ORA_APP_USER" <ora_app_pass>

Enter wallet password: <password>
Create credential oracle.security.client.connect_string1

$ orapki wallet display -wallet $OFFLOAD_HOME/../gluent_wallet -pwd <password>

...
Oracle Secret Store entries:
oracle.security.client.connect_string1
oracle.security.client.password1
oracle.security.client.username1
...

The value for <password> should be replaced with a chosen password to protect the Oracle Wallet.

The value for <ora_app_pass> should be replaced with the unencrypted value of ORA_APP_PASS.

The Oracle Wallet needs to be propagated to all Oracle RAC servers on which Gluent Data Platform is installed.

Oracle PKI Libraries

The following JAR files should be copied from $ORACLE_HOME/jlib to /var/lib/sqoop/ on the node from which Sqoop commands are initiated (OFFLOAD_TRANSPORT_CMD_HOST):

  • oraclepki.jar

  • osdt_cert.jar

  • osdt_core.jar

Copy Oracle Wallet

On the Oracle Database server copy the Oracle Wallet to an appropriate location on the following nodes:

  1. The node from which Sqoop commands are initiated (OFFLOAD_TRANSPORT_CMD_HOST) - must be accessible by OFFLOAD_TRANSPORT_USER

  2. All YARN NodeManager Nodes - must be accessible by OFFLOAD_TRANSPORT_USER when Kerberos is in use or by yarn in non-Kerberos environments

OS User Profile

On the node on which Sqoop commands are initiated, perform the following actions as the OFFLOAD_TRANSPORT_USER:

$ cat << EOF >> ~/.bashrc
export HADOOP_OPTS="-Doracle.net.wallet_location=<full_path_to_wallet> \${HADOOP_OPTS}"
EOF

The value for <full_path_to_wallet> should be replaced with the Oracle Wallet location on the Hadoop nodes from Copy Oracle Wallet.

Update Gluent Environment File

Set OFFLOAD_TRANSPORT_AUTH_USING_ORACLE_WALLET to true in offload.env.

Set SQOOP_OVERRIDES to "-libjars /var/lib/sqoop/oraclepki.jar,/var/lib/sqoop/osdt_cert.jar,/var/lib/sqoop/osdt_core.jar -Dmapreduce.map.java.opts='-Doracle.net.wallet_location=<full_path_to_wallet>'" in offload.env.

The value for <full_path_to_wallet> should be replaced with the Oracle Wallet location on the Hadoop nodes from Copy Oracle Wallet.

Important

Any changes made to the Gluent Data Platform environment file (offload.env) must be propagated across all installations.

Documentation Feedback

Send feedback on this documentation to: feedback@gluent.com