Symptom
-
You are running into errors in the commander.log:
*Exception: InvalidPasskey: The passkey fingerprint ('xx') stored in the database is different from the value in the server passkey file ('yy'). To preserve stored credentials, restore the original passkey file and restart the server. To discard stored credentials and save the new passkey value in the database, update the database configuration with the ignorePasskeyMismatch flag set before connecting to the database.*
-
You are seeing the above message while upgrading CloudBees CD (CloudBees Flow) server to 9.2 from an earlier version.
Issue
The presence of credential data in your database that was previously imported against a different passkey is why you ran into this upgrade blocking event. CloudBees CD (CloudBees Flow) v9.2 enhanced passkey encryption using 256-bit AES (previously 128-bit AES).
Resolution
The hotfix described here can address this problem:
a) Take down all servers in normal order
b) Working with the primary CloudBees CD (CloudBees Flow) server machine, ensure that the connection to the DB is intentionally blocked.
This can be accomplished in many ways. Here are just a few options:
i). Restrict network access, for example, port blocking.
ii). Disable DB user access
iii). Turn the DB off
iv). Use ecconfiguration to point to a non-existent machine
Port blocking seems the easiest on Linux, so we will use this method as an example:
# iptables -A OUTPUT -p tcp --destination-port [port number] -j DROP
Unless you have your own, default Port numbers to use would typically be:
MySQL - 3306
Oracle -1521
MS-SQL - 1433
Verify using telnet that the DB port is not accessible from the CloudBees CD (CloudBees Flow) server .
c) Perform the upgrade, which will install all new bits, but will then face a connection failure to the Database because of (b)
d) Turn the server off using /etc/init.d/commanderServer stop
e) Apply the hotfix found inside of the zip file ec_9.1.0-CEV-24204.zip on Share-file.
f) Re-establish the ability to connect to the database (inverse of what you did in (b) )
# iptables -A OUTPUT -p tcp --destination-port [port number] -j ACCEPT
or you can delete the firewall rule:
iptables -D OUTPUT -p tcp --destination-port [port number] -j ACCEPT
g) Make sure to save the rules and make them permanent.
service iptables save
h) Turn the server on using: /etc/init.d/commanderServer start
-
Test to ensure you are able to login after the startup completes using either the UI or the command line:
eg: ectool --server <servername> login <username> <password>
j) Best-practice recommendation: backup and Replace the passkey and database.properties file on all flow server nodes in the cluster:
cd /opt/electriccloud/electriccommander/conf /opt/electriccloud/electriccommander/jre/bin/java -DCOMMANDER_ZK_CONNECTION=10.0.0.23:2181 -jar /opt/electriccloud/electriccommander/server/bin/zk-config-tool-jar-with-dependencies.jar com.electriccloud.commander.zkconfig.ZKConfigTool --readFile /commander/conf/database.properties database.properties /opt/electriccloud/electriccommander/jre/bin/java -DCOMMANDER_ZK_CONNECTION=10.0.0.23:2181 -jar /opt/electriccloud/electriccommander/server/bin/zk-config-tool-jar-with-dependencies.jar com.electriccloud.commander.zkconfig.ZKConfigTool --readFile /commander/conf/passkey passkey
| You will have to use the COMMANDER_ZK_CONNECTION value from your own <DATA_DIR>/conf/wrapper.conf |
k) Upgrade each additional server in the normal manner - there is no need to apply the hotfix on these extra machines as the hotfix simply addresses an operation at the time of DB Schema upgrade and Passkey update
Your upgrade will no longer block upon seeing credentials which are out of synch with your passkey, but it will record where these are located into your commander.log output files.
Optional Step:
The following command can be used to help identify which credentials may have been contributing to the problem seen yesterday so that your team could hunt them down and re-enter these credentials again. Once this is done with the latest passkey in place, it will ensure all data is properly in synch with the expected passkey file, and no jobs would fail in the future for this same reason.
ls -rt commander-dentsb3utecf01.zip | sort | sed "s/$//" | xargs bash zgrep -h -a "Skipping updating password for credential\|Skipping updating password for Directory Provider\|Skipping updating password for DevOps Insight Server configuration\|Skipping updating password for email configuration\|Corrupted credentials:\|Can not decrypt user" > list_of_things_that_cannot_be_decrypted_in_92_DB.txt