AES-encoded passkey was accidentally overwritten

Description

In case you accidentally overwrote the only copy of the AES-encoded passkey file with another passkey file, you may see the following error message in the server logs:

java.security.GeneralSecurityException: Couldn't decrypt password; the decryption key is likely invalid.

This usually means the server passkey file used to encrypt the password has changed and cannot be used to decrypt the password. If you full exported data from one machine and full imported it to another, then encrypted passwords cannot be accessed on the new machine unless you also copy the server passkey
file across at the same time.

at com.electriccloud.crypto.PasswordCrypterImpl.decrypt(PasswordCrypterImpl.java:234)
at com.electriccloud.security.SessionManagerImpl.decryptSecretId(SessionManagerImpl.java:525)
at com.electriccloud.security.SessionManagerImpl.deleteSessionEntity(SessionManagerImpl.java:1420)
at com.electriccloud.security.SessionManagerImpl.deleteSession(SessionManagerImpl.java:617)▼

Workaround

In case you accidentally overwrote the only copy of the passkey and do not have another instance of the file backed up elsewhere, either on another server or, for clustered configurations, in Zookeeper, the only option you have is to regenerate the passkey manually using the following steps. Following are recovery instructions for both Standalone configurations and Clustered configurations.

Standalone configurations

Delete or rename the passkey file at <DATA_DIR>/conf/passkey.
Change COMMANDER_DB_PASSWORD to plain text in <DATA_DIR>/conf/database.properties.
Start the CloudBees CD/RO server and sign in to the UI. This generates a new passkey file and encrypts the COMMANDER_DB_PASSWORD in the <DATA_DIR>/conf folder using the new AES-256 passkey.
Stop the CloudBees CD/RO server.
Now, sign in to the CloudBees CD/RO UI and manually update the passwords that you saved:
- Any credentials you created
- Passwords for email configuration
- DirectoryProvider passwords
- Analytics configuration password
  
  You are not able to change the SessionID passwords used at runtime for job steps. So you need to rerun those jobs.
As a best practice, back up the new working AES 256 passkey and database.properties.

Clustered configurations

Identify the machine you need to work on and shut down the other machines in the CloudBees CD/RO cluster.

Download database.properties, keystore, passkey, and commander.properties configuration files from ZooKeeper.

cd <install-dir>/conf

<install-dir>/jre/bin/java \
            -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server:port> \
            -jar <install-dir>/server/bin/zk-config-tool-jar-with-dependencies.jar \
            com.electriccloud.commander.zkconfig.ZKConfigTool \
            --readFile /commander/conf/database.properties database.properties

<install-dir>/jre/bin/java \
            -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server:port> \
            -jar <install-dir>/server/bin/zk-config-tool-jar-with-dependencies.jar \
            com.electriccloud.commander.zkconfig.ZKConfigTool \
            --readFile /commander/conf/passkey passkey

<install-dir>/jre/bin/java \
            -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server:port> \
            -jar <install-dir>/server/bin/zk-config-tool-jar-with-dependencies.jar \
            com.electriccloud.commander.zkconfig.ZKConfigTool \
            --readFile /commander/conf/keystore keystore

<install-dir>/jre/bin/java \
            -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server:port> \
            -jar <install-dir>/server/bin/zk-config-tool-jar-with-dependencies.jar \
            com.electriccloud.commander.zkconfig.ZKConfigTool \
            --readFile /commander/conf/commander.properties commander.properties▼

Switch CloudBees CD/RO from cluster mode to standalone mode.
1. To the end of <DATA_DIR>/conf/wrapper.conf file add the following lines:
  wrapper.java.additional.261=-DCOMMANDER_IGNORE_SERVER_MISMATCH=1 wrapper.java.additional.262=-DCOMMANDER_IGNORE_PASSKEY_MISMATCH=1
  ▼
  Ensure that the numbers 261 and 262 are unique within the file. If not, change the number to a unique value.
2. Run the ecconfigure command to set it to single/standalone mode:
  ecconfigure --serverEnableClusteredMode=0 --skipServiceRestart
  ▼
Delete the current pass key in the <DATA_DIR>/conf/passkey file.
Change COMMANDER_DB_PASSWORD to plain text in <DATA_DIR>/conf/database.properties.
Start the CloudBees CD/RO server and sign in to the UI. This generates a new passkey file and encrypts the COMMANDER_DB_PASSWORD in the <DATA_DIR>/conf/database.properties file using the new AES-256 passkey.
Stop the CloudBees CD/RO server.

Push the new passkey and database.properties to Zookeeper.

cd <install-dir>/conf
<install-dir>/jre/bin/java \
        -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server1-ip> \
        -jar <install-dir>/server/bin/zk-config-tool-jar-with-dependencies.jar com.electriccloud.commander.zkconfig.ZKConfigTool \
        --databasePropertiesFile database.properties  \
        --passkeyFile passkey▼

Use COMMANDER_ZK_CONNECTION from your <DATA_DIR>/conf/wrapper.conf file.

Switch the CloudBees CD/RO environment back to cluster mode:

ecconfigure --serverEnableClusteredMode=1 --skipServiceRestart▼

Start the CloudBees CD/RO server.

Download the new working AES-256 passkey and database.properties from Zookeeper and back them up for safe keeping.

<install-dir>/jre/bin/java \
            -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server1-ip> \
            -jar <install-dir>/server/bin/zk-config-tool-jar-with-dependencies.jar com.electriccloud.commander.zkconfig.ZKConfigTool \
            --readFile /commander/conf/database.properties mynew92_aes256_database.properties
<install-dir>/jre/bin/java \
            -DCOMMANDER_ZK_CONNECTION=<zooKeeper-server1-ip> -jar \
            /opt/cloudbees/sda/server/bin/zk-config-tool-jar-with-dependencies.jar com.electriccloud.commander.zkconfig.ZKConfigTool \
            --readFile /commander/conf/passkey mynew92_aes256_passkey▼

Remove the lines you added previously to the end of the <DATA_DIR>/conf/wrapper.conf file:

wrapper.java.additional.261=-DCOMMANDER_IGNORE_SERVER_MISMATCH=1
wrapper.java.additional.262=-DCOMMANDER_IGNORE_PASSKEY_MISMATCH=1▼

Now, sign in to the CloudBees CD/RO UI and manually update the passwords that you saved:
- Any credentials you created
- Passwords for email configuration
- DirectoryProvider passwords
- Analytics configuration password
  
  You are not able to change the SessionID passwords used at runtime for job steps, so you need to rerun those jobs.
Start up the other CloudBees CD/RO servers in the cluster.