Configure custom CAs and CRLs in clustered environments

Configure and integrate custom certificate authorities (CAs) and certificate revocation lists (CRLs) to enhance cyber security, ensure the authenticity of digital certificates, and identify and revoke unauthorized certificates. Although there are several ways to approach the configuration of custom CAs and CRLs in a clustered environment, these instructions offer careful consideration for security and a reliable outcome.

This information only applies to CloudBees CD/RO server nodes in clustered environments.

This process includes these steps:

Verify the ZooKeeper connection
Scale down the system to a single node
Apply changes on the single node to configure the custom certificates
Upload files to ZooKeeper
Change the node to clustered mode

Before performing any changes, back up your data and any relevant configuration files. Failure to back up your files could result in data loss.

Verify the ZooKeeper connection

Follow these steps to verify the ZooKeeper connection.

Change the directory to the CloudBees directory.
```
cd /opt/cloudbees/sda▼
```
Search for COMMANDER_ZK_CONNECTION within the configuration file.
```
grep "COMMANDER_ZK_CONNECTION" ./conf/wrapper.conf▼
```

Check the ZooKeeper connection.

# COMMANDER_ZK_CONNECTION is required for clustered mode
wrapper.java.additional.600=-DCOMMANDER_ZK_CONNECTION=ZooKeeper_Server:2181▼

Scale down the system to a single node

Follow the steps to scale down your system temporarily to a single node.

Select a node to act as the single node for this process. This node will be referred to hereafter as the "single-node".
Log in to the single node with the system user account used in CloudBees CD/RO.
Perform a backup of the ./conf folder on the single node.
Download the Zookeeper files into the single node ./conf folder; begin by changing the directory to cloudbees/sda.
```
cd /opt/cloudbees/sda▼
```

Download the keystore file.

./jre/bin/java \
-DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
-jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
--readFile /commander/conf/keystore ./conf/keystore▼

Download the commander.properties file.

./jre/bin/java \
-DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
-jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
--readFile /commander/conf/commander.properties ./conf/commander.properties▼

Download the security folder.

./jre/bin/java \
-DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
-jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
--readFolder /commander/conf/security ./conf/security▼

Download the passkey file.

./jre/bin/java \
-DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
-jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
--readFile /commander/conf/passkey ./conf/passkey▼

Download the database.properties file.

./jre/bin/java \
-DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
-jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
--readFile /commander/conf/database.properties ./conf/database.properties▼

Backup the ./conf folder to create Zookeeper Backup files. You can use the folder content to restore ZooKeeper to the original state, if necessary.

Before performing any changes, back up your ZooKeeper data and any relevant configuration files. Failure to back up your files could result in data loss.
Stop all the nodes in the cluster, including the single node.
Change the current node to single-node execution by running this command:
```
./bin/ecconfigure --serverEnableClusteredMode=0▼
```
This starts the CloudBees CD/RO Server in non-clustered mode. If the commanderServer service is not automatically started, which could happen if the user account doesn’t have adequate permissions, start it manually.

Once the system starts, it is running in non-clustered mode, with all the files in the local ./conf folder.
Check the system to ensure it is working properly in the non-clustered mode with all the files in the local ./conf folder.
List the certificates on the server; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/keystore▼
```

Check the output; the example below shows there are two certificates.

Your keystore contains 2 entries

ca:server-chained-ca.c.chainpath, Feb 17, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 97:4F:EE:2A:CA:99:23:8A:7D:01:C8:AE:6A:95:AF:17:58:03:34:6F:D0:91:12:62:DC:3B:14:C5:C4:5F:A5:A1
jetty, Feb 17, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 13:15:6A:9A:05:43:B2:47:00:C9:B9:AD:E6:6C:6B:29:A9:3F:A7:CA:83:E9:4A:2B:D6:15:BD:8C:5A:EF:47:21▼

Continue the configuration to prepare the new custom certificates.

Apply changes on the single node to configure the custom certificates

The following files must be available for all affected nodes before proceeding with the configuration steps.

All parent Certificate Authorities (CAs) in PEM format, to generate the chain of trust
The server certificates in PEM format
The server certificate key in PEM format

Convert all certificate files to PEM format

Use the sample code below to convert your files to PEM format.

Convert a DER file (.crt, .cer, .der) to PEM:

openssl x509 -inform der -in certificate.cer -out certificate.pem▼

Convert a PKCS#12 file (.pfx .p12) containing a private key and certificates to PEM:

This format contains all the certificate chains and private key. This ensures the output for the next command is a valid input to use for importing the custom certificate bundle.
```
openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes▼
```
- To output only the private key, add -nocerts to the code above.
- To output only the certificates, add -nokeys to the code above.

Deactivate the default CA key

The default CloudBees CD/RO CAs are not used to sign custom certificates; therefore they need to be deactivated on the server. This is achieved by renaming both of the following files.

Rename the ./conf/security/ca.pem file:

mv ./conf/security/ca.pem ./conf/security/ca.pem_bkp▼

Rename the ca_pk.pem file:

mv ./conf/security/ca_pk.pem ./conf/security/ca_pk.pem_bkp▼

Continue the process by clearing out the keystore.

Remove the default keystore entries

To use custom certificates, the default keystore entries must be cleaned or removed from the server.

List the certificates on the server; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/keystore▼
```

Check the output; the example below shows there are two certificates.

Your keystore contains 2 entries

ca:server-chained-ca.c.chainpath, Feb 17, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 97:4F:EE:2A:CA:99:23:8A:7D:01:C8:AE:6A:95:AF:17:58:03:34:6F:D0:91:12:62:DC:3B:14:C5:C4:5F:A5:A1
jetty, Feb 17, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 13:15:6A:9A:05:43:B2:47:00:C9:B9:AD:E6:6C:6B:29:A9:3F:A7:CA:83:E9:4A:2B:D6:15:BD:8C:5A:EF:47:21▼

Delete the certificates:

./jre/bin/keytool -delete -keystore ./conf/keystore -alias jetty▼

./jre/bin/keytool -delete -keystore ./conf/keystore -alias ca:server-chained-ca.c.chainpath▼

List the certificates on the server to ensure they are removed; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/keystore▼
```
Check the output, it should be similar to:
```
Your keystore contains 0 entries▼
```
Continue the process by trusting the custom certificate.

Trust the custom certificate Root CA

Use these steps to establish trust with the CA chain in the keystore by importing the root CA into the keystore.

Import the root CA in PEM format to the keystore, and trust the certificate; you will be prompted to trust the certificate:
```
./jre/bin/keytool -importcert -file ~/rootCA.pem -keystore ./conf/keystore -alias "CA:root"
...▼
```
Respond to the prompt.
```
Trust this certificate? [no]:  yes▼
```
Check the output; it should look something like:
```
Certificate was added to keystore▼
```
Continue the process by creating the custom certificate bundle.

Create custom certificate bundles

In this section, you will create a certificate bundle for the server.

These files must be available for the server before starting the process:

The certificate in PEM format
- In instances where certificates have parent CAs, the chained CA certificates (in PEM format) should also be added to the certificate to generate the chain of trust.
The certificate key in PEM format

Be sure all files are in PEM format before proceeding; perform these steps to create the custom certificate bundle for the server.

Merge all of the PEM formatted certificates into one file, in the following order:
- Certificate content
  - This is unique to the server.
- Certification authorities:
  - parent CA cert
  - Next CA certs
  - Root CA cert - signed by a third-party trusted CA; or if self-signed, with no parent CA
- Certificate RSA Private Key

Combine the files using the following command, replacing the file names with your file names:

cat CDServerCertificate.pem intermediateCA1.pem intermediateCAx.pem rootCA.pem serverKey.pem > fullChainedCert.pem▼

Check the results. The output file structure should be similar to the following:

- - BEGIN CERTIFICATE - - -    CD Server Certificate
MIIDejCCAmKgAwIBAgIBATANBgkqhkiG9wOBAQsFADBjMRgwFgYDVQQDDA9JbnRI
cm1lZGUhdGUgK0ExFDASBgNVBAsMDFVUZ2luZwVyaW5nMRIWEAYDVQQKDALDbG91
ZEJLZXMXEDAOBgNBAgMB1NldmlsbGExCzAJBgNVBAYTAKVTMB4XDTIZMDIXNZEz
. . .
- - - END CERTIFICATE - - - -
- - BEGIN CERTIFICATE - - -  CD Server Certificate Intermediate CA
MIID2DCCASCgAWIBAgIBATANBgkqhkiG9w0BAOSFADCBkzELMAkGA1UEBhMCRVMX
EDAOBONVBAgMB1NldmlsbGE×EDAOBNVBAcMB1VtYnJldGUXEjAQBgNVBARSCUNS
b3VkOmVlczEUMFIGA1UECwwLRW5naw51ZXJpbmc×EDAOBgNVBAMMB1Jvb30900EX
. . .
- - END CERTIFICATE - - -
- - BEGIN CERTIFICATE - - -  CD Server Certificate Root CA
MIIECTCCAVGgAWIBAgIUKJYtoT6г3iNWOKADTVKUCbAE9mgwDQYJKOZIhvcNAQEL
BQAw9ZM×CZAJBgNVEAYTAKVTMRAWD9YDVOQIDAdTEDZpbGxhMRAWD9YDVOOHDAdV
bWJyZXRIMRIWEAYDVOOKDALDbG91ZEJlZXMXFDASB9NVBASMCOVUZ21uZWVyaW5n
. . .
- - END CERTIFICATE - - -
- - BEGIN RSA PRIVATE KEY - - -  CD Server Certificate Private Key
MIIEOgIBAAKCAQG5ZWVTFKboPPOdWkfkg085neMa59xigKw/fUNObYkyDsJa4Q1g
fUSAa+ZVQ6ot×J4rVvwaN15R2UEMn01oGf2rAYZm1Faun5RnQ1ALHtHEdUkg466c
DzOaFmYjFOTqcFrT00o2mjGtIKACg+oz+JflfObbjeyeG8EBgnyq8L9i7C1n09Gy
. . .
- - - END RSA PRIVATE KEY - - -▼

Continue the process to import the custom certificate bundles.

Import the custom certificate bundles

This section explains how to create a new temporary keystore containing only a custom certificate bundle for the server. The command also sets a password for the keystore to prevent null errors. If the certificate’s private key is encrypted, the system prompts you to enter the private key passphrase. Each certificate bundle must be imported to the server.

Create the temporary keystore on the server:

openssl pkcs12 \
        -export \
        -name jetty \
        -in fullChainedCert.pem \ << Generated in the previous step
        -out ./conf/keystore_temp \
        -passout pass:xxxxxxxx▼

Run the code below to migrate the jetty certificate from the temp keystore on the server to the CloudBees CD/RO keystore:

./jre/bin/keytool \
        -importkeystore \
        -srckeystore ./conf/keystore_temp \
        -srcstorepass xxxxxx \
        -srcalias jetty \
        -destkeystore ./conf/keystore \
        -deststorepass xxxxxxxx \
        -destalias jetty \
        -deststoretype JKS▼

Delete the temporary keystore on the server:
```
rm -f ./conf/keystore_temp▼
```
Restart the CloudBees CD/RO server.

Non-encrypted private keys and certificate bundles containing private keys should be removed from the system or stored in a secure folder.
```
./bin/eccert list --server▼
```

The response should be similar to the following:

Server keystore:

    Trusted CA 'root'
        EMAILADDRESS=email@company.com, CN=Root CA, OU=Engineering, O=Company, L=Umbrete, ST=Sevilla, C=ES (index: 28955da13ezede2356d0a0034d529409b004f668)

    Server
        O=server, CN=server-chained-ca (index: 1)
        C=ES, ST=Sevilla, O=Company OU=Engineering, CN=Intermediate CA (index: 1)
        EMAILADDRESS=email@company.com, CN=Root CA, OU=Engineering, O=Company L=Umbrete, ST=Sevilla, C=ES (index: 28955da13ezede2356d0a0034d529409b004f668)▼

Continue the process by setting up the custom CRL chain.

Configure the custom CRL for each CA

CloudBees CD/RO includes a default CRL file to verify certificate revocation for the default autogenerated certificates. For custom certificates, CloudBees CD/RO supports three different methods for providing certificate revocation lists.

Select a method:
- Specifying a PEM file in the local file system.
- Specifying a URL to the CRL distribution point.
- Relying on the CRL distribution points attribute for the CloudBees CD/RO server certificate.
Choose the property values.

The values in these properties must be separated by a semicolon (;). The acceptable values are:
- A list of file paths to the PEM files in the local file system
- A list of URLs to the CRL distribution point
- A mix of both

The previous property COMMANDER_CRL_FILE has been deprecated in favor of the new property: COMMANDER_CRL_URL. Begin using the new property now and comment out the old property.

Configure the COMMANDER properties.
- Configure the server property by:
  1. Opening this file: ./conf/commander.properties
  2. Configuring this property: COMMANDER_CRL_URL
    
    COMMANDER_CRL_URL=conf/security/crl_rootCA.pem; http://ca-cd-server:9090/ca.pem
    ▼
  3. Restart the server.
Continue the process in one of the following ways:
- If you are relying on the certificate CRL distribution point attribute for the CloudBees CD/RO server continue to the Establish the CRL distribution point.
- Otherwise, continue to the Specify the CRL certificate refresh section.

Establish the CRL distribution point

To use the CRL distribution points included in custom certificates, the new property COMMANDER_CRL_URL and the old property COMMANDER_CRL_FILE need to be absent or empty.

When the properties are empty, the server will automatically search for distribution points within the keystore, specifically examining the certificate chain of the "jetty" alias.

You must restart the server for the changes in these properties to take effect.

Specify the CRL certificate refresh

A new setting has been added to the CloudBees CD/RO server, CRL certificate reload frequency in days. Follow the steps to set the certificate refresh rate.

Go to Administration Security settings.
Set the number (in days) for the CRL certificate refresh.

This setting enables you to specify how often the server will automatically refresh CRL certificates. The default value is set to 30 days and the value is stored in the server property crlCertificateReloadFrequency.

Figure 1. Server settings
Run a few tests to ensure your system is working properly.
Continue the process to push the custom configuration files to ZooKeeper.

Upload files to ZooKeeper

This section provides the steps to upload the configuration changes that you made to the single node, to ZooKeeper.

Upload the keystore folder to ZooKeeper.

./jre/bin/java \
    -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
    -jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
    --keystoreFile ./conf/keystore▼

Upload commander.properties to ZooKeeper.

./jre/bin/java \
    -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
    -jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
    --commanderPropertiesFile ./conf/commander.properties▼

Upload the security folder to ZooKeeper.

./jre/bin/java \
    -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server>:2181 \
    -jar ./server/bin/zk-config-tool-jar-with-dependencies.jar \
    --writeFolder /commander/conf/security ./conf/security▼

Continue the process to change the single node back to clusterd mode.

Change the node to clustered mode

With all files and folders now in ZooKeeper, you can change the single node back to clustered mode.

Run the command to enable clustered mode:
```
./bin/ecconfigure --serverEnableClusteredMode=1▼
```
This restarts the CloudBees CD/RO server in clustered mode.

If the service is not automatically restarted, which may occur if the user account does not have adequate permissions, restart it manually.

The system should now be running in cluster mode with the configuration files from Zookeeper.

Validate the certificate in this node is correct, and start all the other nodes.

Once restarted, all nodes will use the same certificate, signed by the CA.

Additional resources

If you are using trusted agents, refer to Configure custom CAs and CRLs in a non-clustered environment.

For more information about Zookeeper refer to Upload configuration files to ZooKeeper.