Configure custom CAs and CRLs in non-clustered environments

Configure and integrate custom certificate authorities (CAs) and certificate revocation lists (CRLs) to enhance cyber security, ensure the authenticity of digital certificates, and identify and revoke unauthorized certificates.

These instructions apply to non-clustered environments and trusted agents.

Before doing anything in this section, copy the ./conf folder for all servers and trusted agents you plan to apply custom certificates. Failure to back up these ./conf files could result in data loss.

The process of configuring your custom CAs and CRLs, includes these key tasks:

Reviewing the prerequisite requirements.
Deactivating the default certificates.
Removing the default keystore entries.
Trusting the new custom certificates.
Creating the new custom certificate bundle.
Importing the custom certificate bundle.
Configuring custom CAs and CRLs.

Prerequisites

The following files must be available for the server and trusted agents where the custom certificates will be applied.

All parent Certificate Authorities (CAs) in PEM format, to generate the chain of trust
The server/trusted agent certificates in PEM format
The server/trusted agent certificate key in PEM format

Use the sample code below to convert your files to PEM format.

Convert a DER file (.crt, .cer, .der) to PEM:

openssl x509 -inform der -in certificate.cer -out certificate.pem▼

Convert a PKCS#12 file (.pfx .p12) containing a private key and certificates to PEM:

This format contains all the certificate chains and private key. This ensures the output for the next command is a valid input to use for importing the custom certificate bundle.
```
openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes▼
```
- To output only the private key, add -nocerts to the code above.
- To output only the certificates, add -nokeys to the code above.

Deactivate the default CA key

The default CloudBees CD/RO CAs are not used to sign custom certificates; therefore they need to be deactivated on the server. This is achieved by renaming both of the following files.

Rename the ./conf/security/ca.pem file:

mv ./conf/security/ca.pem ./conf/security/ca.pem_bkp▼

Rename the ca_pk.pem file:

mv ./conf/security/ca_pk.pem ./conf/security/ca_pk.pem_bkp▼

Continue the process by clearing out the keystore.

Remove the default keystore entries

To use custom certificates, the default keystore entries must be cleaned or removed from the server and trusted agents.

Server

List the certificates on the server; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/keystore▼
```

Check the output; the example below shows there are two certificates.

Your keystore contains 2 entries

ca:server-chained-ca.c.chainpath, Feb 17, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 97:4F:EE:2A:CA:99:23:8A:7D:01:C8:AE:6A:95:AF:17:58:03:34:6F:D0:91:12:62:DC:3B:14:C5:C4:5F:A5:A1
jetty, Feb 17, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 13:15:6A:9A:05:43:B2:47:00:C9:B9:AD:E6:6C:6B:29:A9:3F:A7:CA:83:E9:4A:2B:D6:15:BD:8C:5A:EF:47:21▼

Delete the certificates:

./jre/bin/keytool -delete -keystore ./conf/keystore -alias jetty▼

./jre/bin/keytool -delete -keystore ./conf/keystore -alias ca:server-chained-ca.c.chainpath▼

List the certificates on the server to ensure they are removed; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/keystore▼
```
Check the output, it should be similar to:
```
Your keystore contains 0 entries▼
```

Trusted agent

List the certificates on the trusted agent; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/agent/keystore▼
```

Check the output; the example below shows there are two certificates.

Your keystore contains 2 entries

ca:server-chained-ca.c.chainpath, Feb 17, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 97:4F:EE:2A:CA:99:2D:8A:73:03:C8:AE:6A:95:AF:11:58:03:31:6F:D0:97:12:62:DC:3B:14:C5:C4:5F:A5:A1
jetty, Feb 17, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 13:15:6A:99:05:43:BD:47:00:C9:B9:AD:E6:6C:65:27:A2:3F:A7:CA:83:E9:4A:23:D6:15:BD:8C:5A:EF:47:21▼

Delete the default certificates:

./jre/bin/keytool -delete -keystore ./conf/agent/keystore -alias jetty▼

./jre/bin/keytool -delete -keystore ./conf/agent/keystore -alias ca:server-chained-ca.c.chainpath▼

List the certificates on the trusted agents to make sure they are removed; if prompted, enter the keystore password.
```
./jre/bin/keytool -list -keystore ./conf/agent/keystore▼
```
Check the outpput, it should be similar to:
```
Your keystore contains 0 entries▼
```
Continue the process by trusting the custom certificate.

Trust the custom certificate Root CA

Use these steps to establish trust with the CA chain in the keystore by importing the root CA into the keystore.

Server

Import the root CA in PEM format to the keystore, and trust the certificate; you will be prompted to trust the certificate:
```
./jre/bin/keytool -importcert -file ~/rootCA.pem -keystore ./conf/keystore -alias "CA:root"
...▼
```
Respond to the prompt.
```
Trust this certificate? [no]:  yes▼
```
Check the output; it should look something like:
```
Certificate was added to keystore▼
```

Trusted agent

Import the root CA in PEM format to the keystore and trust the certificate; you will be prompted to trust the certificate:
```
./jre/bin/keytool -importcert -file ~/rootCA.pem -keystore ./conf/agent/keystore -alias "CA:root"▼
```

Respond to the prompt.

...
Trust this certificate? [no]:  yes▼

Check the output, it should look something like this:
```
Certificate was added to keystore▼
```
Continue the process by creating the custom certificate bundle.

Create custom certificate bundles

In this section, you will create a certificate bundle for the server and a certificate bundle for each trusted agent.

These files must be available for the server and the trusted agents before starting the process:

The certificate in PEM format
- In instances where certificates have parent CAs, the chained CA certificates (in PEM format) should also be added to the certificate to generate the chain of trust.
The certificate key in PEM format

Server

Be sure all files are in PEM format before proceeding; perform these steps to create the custom certificate bundle for the server.

Merge all of the PEM formatted certificates into one file, in the following order:
- Certificate content
  - This is unique to the server.
- Certification authorities:
  - parent CA cert
  - Next CA certs
  - Root CA cert - signed by a third-party trusted CA; or if self-signed, with no parent CA
- Certificate RSA Private Key

Combine the files using the following command, replacing the file names with your file names:

cat CDServerCertificate.pem intermediateCA1.pem intermediateCAx.pem rootCA.pem serverKey.pem > fullChainedCert.pem▼

Check the results. The output file structure should be similar to the following:

- - BEGIN CERTIFICATE - - -    CD Server Certificate
MIIDejCCAmKgAwIBAgIBATANBgkqhkiG9wOBAQsFADBjMRgwFgYDVQQDDA9JbnRI
cm1lZGUhdGUgK0ExFDASBgNVBAsMDFVUZ2luZwVyaW5nMRIWEAYDVQQKDALDbG91
ZEJLZXMXEDAOBgNBAgMB1NldmlsbGExCzAJBgNVBAYTAKVTMB4XDTIZMDIXNZEz
. . .
- - - END CERTIFICATE - - - -
- - BEGIN CERTIFICATE - - -  CD Server Certificate Intermediate CA
MIID2DCCASCgAWIBAgIBATANBgkqhkiG9w0BAOSFADCBkzELMAkGA1UEBhMCRVMX
EDAOBONVBAgMB1NldmlsbGE×EDAOBNVBAcMB1VtYnJldGUXEjAQBgNVBARSCUNS
b3VkOmVlczEUMFIGA1UECwwLRW5naw51ZXJpbmc×EDAOBgNVBAMMB1Jvb30900EX
. . .
- - END CERTIFICATE - - -
- - BEGIN CERTIFICATE - - -  CD Server Certificate Root CA
MIIECTCCAVGgAWIBAgIUKJYtoT6г3iNWOKADTVKUCbAE9mgwDQYJKOZIhvcNAQEL
BQAw9ZM×CZAJBgNVEAYTAKVTMRAWD9YDVOQIDAdTEDZpbGxhMRAWD9YDVOOHDAdV
bWJyZXRIMRIWEAYDVOOKDALDbG91ZEJlZXMXFDASB9NVBASMCOVUZ21uZWVyaW5n
. . .
- - END CERTIFICATE - - -
- - BEGIN RSA PRIVATE KEY - - -  CD Server Certificate Private Key
MIIEOgIBAAKCAQG5ZWVTFKboPPOdWkfkg085neMa59xigKw/fUNObYkyDsJa4Q1g
fUSAa+ZVQ6ot×J4rVvwaN15R2UEMn01oGf2rAYZm1Faun5RnQ1ALHtHEdUkg466c
DzOaFmYjFOTqcFrT00o2mjGtIKACg+oz+JflfObbjeyeG8EBgnyq8L9i7C1n09Gy
. . .
- - - END RSA PRIVATE KEY - - -▼

Trusted agent

Be sure all files are in PEM format before proceeding; perform these steps to create the custom certificate bundle for each trusted agent.

Merge all of the PEM formatted certificates into one file, in the following order:
- Certificate content
  - This is unique to each trusted agent.
- Certification authorities:
  - parent CA cert
  - Next CA certs
  - Root CA cert - signed by a third-party trusted CA; or if self-signed, with no parent CA
- Certificate RSA Private Key

Combine the files using the following command, replacing the file names with your file names:

cat CDServerCertificate.pem intermediateCA1.pem intermediateCAx.pem rootCA.pem serverKey.pem > fullChainedCert.pem▼

Check the results. The output file structure should be similar to the following:

- - BEGIN CERTIFICATE - - -    CD Trusted Agent Certificate
MIIDejCCAmKgAwIBAgIBATANBgkqhkiG9wOBAQsFADBjMRgwFgYDVQQDDA9JbnRI
cm1lZGUhdGUgK0ExFDASBgNVBAsMDFVUZ2luZwVyaW5nMRIWEAYDVQQKDALDbG91
ZEJLZXMXEDAOBgNBAgMB1NldmlsbGExCzAJBgNVBAYTAKVTMB4XDTIZMDIXNZEz
. . .
- - - END CERTIFICATE - - - -
- - BEGIN CERTIFICATE - - -  CD Trusted Agent  Certificate Intermediate CA
MIID2DCCASCgAWIBAgIBATANBgkqhkiG9w0BAOSFADCBkzELMAkGA1UEBhMCRVMX
EDAOBONVBAgMB1NldmlsbGE×EDAOBNVBAcMB1VtYnJldGUXEjAQBgNVBARSCUNS
b3VkOmVlczEUMFIGA1UECwwLRW5naw51ZXJpbmc×EDAOBgNVBAMMB1Jvb30900EX
. . .
- - END CERTIFICATE - - -
- - BEGIN CERTIFICATE - - -  CD Trusted Agent  Certificate Root CA
MIIECTCCAVGgAWIBAgIUKJYtoT6г3iNWOKADTVKUCbAE9mgwDQYJKOZIhvcNAQEL
BQAw9ZM×CZAJBgNVEAYTAKVTMRAWD9YDVOQIDAdTEDZpbGxhMRAWD9YDVOOHDAdV
bWJyZXRIMRIWEAYDVOOKDALDbG91ZEJlZXMXFDASB9NVBASMCOVUZ21uZWVyaW5n
. . .
- - END CERTIFICATE - - -
- - BEGIN RSA PRIVATE KEY - - -  CD Trusted Agent  Certificate Private Key
MIIEOgIBAAKCAQG5ZWVTFKboPPOdWkfkg085neMa59xigKw/fUNObYkyDsJa4Q1g
fUSAa+ZVQ6ot×J4rVvwaN15R2UEMn01oGf2rAYZm1Faun5RnQ1ALHtHEdUkg466c
DzOaFmYjFOTqcFrT00o2mjGtIKACg+oz+JflfObbjeyeG8EBgnyq8L9i7C1n09Gy
. . .
- - - END RSA PRIVATE KEY - - -▼

Continue the process to import the custom certificate bundles.

Import the custom certificate bundles

This section explains how to create a new temporary keystore containing only a custom certificate bundle for the server and a custom certificate bundle for each trusted agent. The command also sets a password for the keystore to prevent null errors. If the certificate’s private key is encrypted, the system prompts you to enter the private key passphrase. Each certificate bundle must be imported on the server and the trusted agents.

Server

Create the temporary keystore on the server:

openssl pkcs12 \
        -export \
        -name jetty \
        -in fullChainedCert.pem \ << Generated in the previous step
        -out ./conf/keystore_temp \
        -passout pass:xxxxxxxx▼

Run the code below to migrate the jetty certificate from the temp keystore on the server to the CloudBees CD/RO keystore:

./jre/bin/keytool \
        -importkeystore \
        -srckeystore ./conf/keystore_temp \
        -srcstorepass xxxxxx \
        -srcalias jetty \
        -destkeystore ./conf/keystore \
        -deststorepass xxxxxxxx \
        -destalias jetty \
        -deststoretype JKS▼

Delete the temporary keystore on the server:
```
rm -f ./conf/keystore_temp▼
```
Restart the CloudBees CD/RO server.

Non-encrypted private keys and certificate bundles containing private keys should be removed from the system or stored in a secure folder.
```
./bin/eccert list --server▼
```

The response should be similar to the following:

Server keystore:

    Trusted CA 'root'
        EMAILADDRESS=email@company.com, CN=Root CA, OU=Engineering, O=Company, L=Umbrete, ST=Sevilla, C=ES (index: 28955da13ezede2356d0a0034d529409b004f668)

    Server
        O=server, CN=server-chained-ca (index: 1)
        C=ES, ST=Sevilla, O=Company OU=Engineering, CN=Intermediate CA (index: 1)
        EMAILADDRESS=email@company.com, CN=Root CA, OU=Engineering, O=Company L=Umbrete, ST=Sevilla, C=ES (index: 28955da13ezede2356d0a0034d529409b004f668)▼

Trusted agent

Create the temporary keystore on each trusted agent:

openssl pkcs12 \
        -export \
        -name jetty \
        -in fullChainedCert.pem \ << Generated in the previous step
        -out ./conf/agent/keystore_temp \
        -passout pass:xxxxxxxx▼

Migrate the jetty certificate from the temporary keystore on the trusted agents to the CloudBees CD/RO keystore:

./jre/bin/keytool \
        -importkeystore \
        -srckeystore ./conf/agent/keystore_temp \
        -srcstorepass xxxxxx \
        -srcalias jetty \
        -destkeystore ./conf/agent/keystore \
        -deststorepass xxxxxxxx \
        -destalias jetty \
        -deststoretype JKS▼

Delete the temporary keystore on the trusted agents:
```
rm -f ./conf/agent/keystore_temp▼
```
Restart the CloudBees CD/RO trusted agent.

Non-encrypted private keys and certificate bundles containing private keys should be removed from the system or stored in a secure folder.
```
./bin/eccert list --agent▼
```

The response should be similar to the following:

Agent keystore:

    Trusted CA 'root'
        EMAILADDRESS=email@company.com, CN=Root CA, OU=Engineering, O=Company, L=Umbrete, ST=Sevilla, C=ES (index: 28955da13ezede2356d0a0034d529409b004f668)

    Agent
        O=agent, CN=server-chained-ca-agent (index: 9)
        C=ES, ST=Sevilla, O=Company, OU=Engineering, CN=Intermediate CA (index: 1)
        EMAILADDRESS=email@company.com, CN=Root CA, OU=Engineering, O=Company, L=Umbrete, ST=Sevilla, C=ES (index: 28955da13ezede2356d0a0034d529409b004f668)▼

Continue the process by setting up the custom CRL chain.

Configure the custom CRL for each CA

CloudBees CD/RO includes a default CRL file to verify certificate revocation for the default autogenerated certificates. For custom certificates, CloudBees CD/RO supports three different methods for providing certificate revocation lists.

Select a method:
- Specifying a PEM file in the local file system.
- Specifying a URL to the CRL distribution point.
- Relying on the CRL distribution points attribute for the CD server / trusted agent certificate.
Choose the property values.

The values in these properties must be separated by a semicolon (;). The acceptable values are:
- A list of file paths to the PEM files in the local file system
- A list of URLs to the CRL distribution point
- A mix of both

The previous properties COMMANDER_CRL_FILE and AGENT_CRL_FILE have been deprecated in favor of the new properties: COMMANDER_CRL_URL and AGENT_CRL_URL. Begin using the new properties now and comment out the old properties.

Configure the COMMANDER and AGENT properties.
- Configure the server property by:
  1. Opening this file: ./conf/commander.properties
  2. Configuring this property: COMMANDER_CRL_URL
    
    COMMANDER_CRL_URL=conf/security/crl_rootCA.pem; http://ca-cd-server:9090/ca.pem
    ▼
  3. Restart the server.
- Configure the properties for all agents by:
  1. Opening this file on each agent: ./conf/agent/agent.properties
  2. Configuring this property: AGENT_CRL_URL
    
    AGENT_CRL_URL=conf/agent/crl_rootCA.pem; http://ca-cd-server:9090/ca.pem
    ▼
  3. Restart the trusted agent.
Continue the process in one of the following ways:
- If you are relying on the certificate CRL distribution point attribute for the CD server / trusted agent, continue to the Using the Certificate CRL Distribution point.
- Otherwise, continue to the Specify CRL certificate refresh section.

Using the Certificate CRL Distribution point

To use the CRL distribution points included in custom certificates, both new properties COMMANDER_CRL_URL and AGENT_CRL_URL and old properties COMMANDER_CRL_FILE and AGENT_CRL_FILE need to be absent or empty.

When the properties are empty, the server or trusted agent will automatically search for distribution points within the keystore, specifically examining the certificate chain of the "jetty" alias.

You must restart the server and trusted agents for the changes in these properties to take effect.

Specify CRL certificate refresh

For the CloudBees CD/RO server, a new setting, CRL certificate reload frequency in days, has been added to Administration Security settings. This setting enables you to specify how often the server will automatically refresh CRL certificates. The default value is set to 30 days and the value is stored in the server property crlCertificateReloadFrequency.

You must manually restart the trusted agents for the changes to take effect.