Installing CloudBees Analytics to an existing CloudBees CI on traditional platforms environment

Installations of CloudBees CI on traditional platforms can take advantage of the analytics capabilities of CloudBees Software Delivery Automation.

Follow these steps in the order listed to complete the installation:

Review the Pre-installation requirements
Upgrade CloudBees CI on traditional platforms to version 2.263.4.2
Install CloudBees Analytics prerequisites
Install CloudBees Analytics

Pre-installation requirements

Before starting the installation process, do the following:

Contact CloudBees Support to obtain the executable files for the CloudBees CD/RO and CloudBees Analytics installers.
Review the CloudBees CI on traditional platforms + CloudBees Analytics architecture details to become familiar with CloudBees CI and CloudBees Software Delivery Automation concepts and terminology.

Upgrading CloudBees CI on traditional platforms

Before upgrading

Before upgrading, take a snapshot of your CloudBees CI platform for the operations center and every controller:

Generate a support bundle. It will help if there are problems with the upgrade or to know what plugins were upgraded in the process.
Backup your instance before starting the upgrade. The minimal backup is a copy of your ${JENKINS_HOME} directory. If the BUILD_DIRECTORY is out of the ${JENKINS_HOME}, you must back it up as well.

If something unexpected occurs, the rollback process is based on the backup of your instance.
Upgrade your plugins before starting the upgrade. Run tests after upgrading the plugins to ensure the upgrades are functioning as expected.

For client controllers that are connected to the operations center:

The operations center must be upgraded first, and then client controllers can be upgraded.
The client controller connected to the operations center does not have to be at the same version. The operations center version must always be more recent or as old as the version of the client controllers that are connected to the operations center (operations center >= client controller version).

Key principles to follow when upgrading

Change one thing at a time

Limit the changes you make to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, or OS package upgrades when upgrading Jenkins. This helps to identify and isolate the root cause of any issues you encounter.

Backup strategies

CloudBees recommends the following backup strategies:

Ensure you have regular backups of your JENKINS_HOME for the operations center and all controllers.

CloudBees recommends using the CloudBees Backup Plugin plugin to automate this. controllers that have the CloudBees Backup Plugin 3.38 or later installed have support for the Restore job type.
- To restore an operations center when using CloudBees Software Delivery Automation, follow the steps for Restoring manually.
  
  The backups should be stored at an offsite location such as cloud storage or, at a minimum, on separate hardware from your CloudBees CI.
Take another backup immediately before you start the upgrade.
Test the backup to verify it is valid.

The backup is used to recover from a failure if you encounter a severe issue during the production upgrade.

After an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins. The only way to roll back is to restore the Jenkins_HOME backup.

To validate the integrity of the backup, it is essential to extract the backup to another filesystem and verify the checksums using:
```
find $JENKINS_HOME -type f | sort | xargs md5sum >~/checksum.log.backup▼
```
Run this for both the live and backup data, and compare the checksum.log.* between the two.

Test strategies

CloudBees recommends the following test strategies:

Create a separate test environment that has a similar setup as your production environment with representative jobs.

If you are creating a test environment and require a license, please send CloudBees the instance ID for your test instance, and CloudBees will send you a test instance license.

Create jobs on your test controller that are representative of jobs on your production controller. This allows you to test the upgrade using the key plugins for your workflow and interactions with any external systems such as SCM tools, artifact repositories, bug trackers, and cloud automation.

It is a best practice to create jobs that are a subset of representative jobs on the test cluster. These dedicated jobs represent your production jobs, but without working on real data. CloudBees recommends talking to your development groups to find jobs that cover the primary functionality, such as:

Test your end-to-end complex pipeline jobs.
Test jobs that utilize critical plugins.
Run jobs that are part of integrations.

Run jobs that are for specific technologies.

To clone a production controller to a test controller, follow the steps from Migration Guide: CloudBees CI, specifically the section titled, Case B - Migrate Entire Jenkins Configuration, Including Jobs.

Update the test configuration for some representative jobs so they do not collide with your production jobs. Then, run the test jobs to ensure they work correctly in the upgraded test instance. You will need to recreate credentials on your test instance.

This test environment can also be useful after the upgrade to test the installation of new plugins.

Recommendations

Set up a test environment

This is ideally a full test environment with the operations center and client controllers and requires a test license. If this is not feasible, using a test controller with representative jobs will work.

Enable the CloudBees Assurance Program (CAP)

Verify CAP is enabled to avoid plugin dependency issues or incorrect versions installed.

Upgrade to the latest version

CloudBees recommends that you always upgrade to the latest version of your CloudBees product, ensuring wider lifecycle support coverage, plus more security patches added via the CloudBees Security Advisory, resolved issues, and new features.

Upgrade the instance where it is running

CloudBees highly recommends that you:

Upgrade Jenkins to the same location where it is currently running, even for production environments.
Perform a backup of $JENKINS_HOME and the $BUILD_DIR, in case it is not in the default location. This should allow you to revert to the previous state.

CloudBees does not recommend that you keep two different instances running at the same time. This may be done to avoid downtime while performing the upgrade. However, unless you really know what you are doing, it is very difficult to replicate the exact same environment. For example:

The JNLP agent does not work correctly as the $JENKINS_URL because the location is different.
Credentials may fail if the secret is not the same for both instances.
The OS may not be configured in the same way. For example, ulimit.
The CloudBees CI configuration may not be correctly replicated.

Stop using Apache Maven builds

The Maven Integration plugin is not recommended because it is considered unreliable by the Jenkins Community.

End-of-life announcement

CloudBees has determined that this plugin will be removed from the CloudBees Assurance Program (CAP) in late 2022. Notification will be provided prior to the removal of the plugin. Please contact CloudBees Support if you have any concerns or questions.

CloudBees recommends that you migrate jobs using the Maven Integration plugin to Pipeline Jobs using the Pipeline Maven Integration plugin.

If you are able to migrate your jobs, please carefully read the Maven jobs and Java versions compatibility guide.

Upgrade process

Visit the downloads page. Select your desired distribution type and review the instructions below.

Java WAR file
Ubuntu/Debian
Red Hat / CentOS
OpenSuse
Microsoft Windows
Docker

Java WAR file

If your CloudBees CI instance is run with the command java -jar cloudbees-core-cm.war or java -jar cloudbees-core-oc.war, you can simply replace the jenkins.war file with the latest version.

Ubuntu/Debian

Operations center - apt-get update && apt-get install cloudbees-core-oc
Controller - apt-get update && apt-get install cloudbees-core-cm

Red Hat/CentOS

Beginning in CloudBees CI on traditional platforms version 2.319.1.5, the epel-release package is required to upgrade CloudBees CI on traditional platforms using an RPM, however some CentOS distributions do not include it. For those CentOS distributions, run the following command before you begin the installation procedure: sudo yum install epel-release.

Operations center - yum upgrade cloudbees-core-oc
Controller - yum upgrade cloudbees-core-cm

RPM/YUM with High Availability: The RPM package contains a post-install script to ensure ownership on several files including Jenkins_HOME. If High Availability is configured, I/O errors can occur. For example, when upgrading one node while another node is still running. For more information, refer to JENKINS-23273.

To address this problem, you can skip this script by adding the property JENKINS_INSTALL_SKIP_CHOWN="true" under /etc/sysconfig/jenkins.

OpenSUSE/SUSE Linux

Operations center - zypper install cloudbees-core-oc
Controller - zypper install cloudbees-core-cm

Microsoft Windows

To upgrade on Microsoft Windows servers, a new .zip must be downloaded from the downloads page and installed.

Servlet container

Once a .war archive has been downloaded, follow the servlet container’s existing application deployment process.

When using servlet containers, CloudBees CI will set the JENKINS_HOME to the $APP_SERVER_USER/.jenkins/ folder. If the servlet container installation does not include write permissions to this folder for this user (sometimes done for security), you must either grant appropriate permissions or override this setting by adding the -DJENKINS_HOME=$MY_JENKINSPATH argument in your servlet container startup. Refer to the servlet container’s documentation for how to add startup arguments.

Tomcat

Use environment variable CATALINA_OPTS to add:

-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true.
-Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true which is needed for the Blue Ocean plugin.

CloudBees recommends that you configure it in the script $CATALINA_BASE/bin/setenv.sh (Linux) or %CATALINA_BASE%\bin\setenv.bat (Microsoft Windows) that you’ll create to customize your application server.

Custom container installations

If you use a custom container, you will find the cloudbees-core-oc.war or cloudbees-core-cm.war file in the deploy directory of your container. For example, /usr/local/jboss/server/default/deploy/cloudbees-core-cm.war is the location for a default JBoss installation.

Upgrade steps

Before you upgrade

Ensure you have enabled the CloudBees Assurance Program (CAP) to ensure you are upgrading to plugin versions tested by CAP. For more information, please refer to Upgrading plugins with Beekeeper Upgrade Assistant.
Enable the CloudBees Quiet Start Plugin plugin to ensure that when the instance first comes online after the upgrade, builds will not start automatically before you get a chance to upgrade the plugins.
Verify the test environment is functioning correctly after it is upgraded.
Ensure you have taken a backup.
Verify all running builds are complete.
Java 11 is recommended for your Jenkins instance, the operations center, Jenkins controllers, and all agents for Jenkins LTS 2.303.3. For more information, refer to the upgrade guide.

CloudBees strongly recommends that you upgrade to Java 11 as soon as possible. The ability to run new versions of the product on Java 8 will be removed in an upcoming release. Releases that occurred during the support window for Java 8 will continue to run on Java 8.

Recommended steps

Upgrade the plugins on the operations center.

Select Manage Jenkins Beekeeper Upgrade Assistant CAP Configuration.
Verify that Enroll this instance in the CloudBees Assurance Program is selected.
Verify that Allow automatic upgrades of plugins on restart is selected.
Select Save.
Navigate to Manage Jenkins Manage Plugins and under the Updates tab, select upgrade all plugins.

Restart the operations center.

If you are using the CAS plugin and upgrading to version 2.277.1.2 or later, do not upgrade it using the Plugin Manager. It must be upgraded at the same time as the instance. Once the service is stopped, you must download the CAS plugin and manually replace $JENKINS_HOME/plugins/cas-plugin.jpi.

After your controllers are upgraded, you must manually upgrade the agent.jar file for any inbound agents that are not configured to upgrade the agent.jar file automatically. For more information, refer to Windows agent offline or unable to connect.
SSH-connected agents upgrade the agent.jar automatically.

You can enable automatic upgrades for inbound agents by Upgrading Windows Service Wrapper.
Now that you are running the latest version of CloudBees CI, CloudBees recommends you enable actionable build notifications in GitHub, Bitbucket, and Slack. This generally takes a few minutes and provides enhanced notifications for your developers with no Pipeline changes required. For more information, refer to Enabling actionable build notifications in GitHub and Bitbucket and Setting up actionable build notifications in Slack.

When updating your controllers, CloudBees recommends that you install the Health Advisor by CloudBees plugin, if not already installed. The CloudBees Jenkins Health Advisor automatically identifies issues that could impact the performance, stability, and security of your controller. It also identifies potential issues due to known issues on your controller. You are notified via email when a new problem is found and the email includes links to the solutions for the identified issues. For instructions, refer to Jenkins Health Advisor by CloudBees.

If you are using the High Availability (HA) feature, the general process for updating the HA operations center or any HA controller is:

Stop both HA nodes.
Upgrade both HA nodes.
Start one of the HA nodes and wait for it to come fully online.
Update the plugins according to CAP recommendations and restart.
Start the other HA node.

Post-upgrade notes

Review articles in our Best Practices and apply changes accordingly.

Installing CloudBees Analytics prerequisites

The next step is to install the prerequisites for CloudBees Analytics, which include the CloudBees CD/RO server, a webserver, a local agent, and a MariaDB database.

The artifact repository is not installed. Also, when installing the CloudBees CD/RO server, the CloudBees CD/RO plugins are not installed.

Use the steps in this section to install the prerequisite CloudBees Software Delivery Automation components needed for CloudBees Analytics. Review Before You Install CloudBees Software Delivery Automation before performing this procedure.

The built-in database is not supported in a clustered CloudBees Software Delivery Automation configuration.

(Linux only) Enter the following command to make the installer file executable:
```
chmod +x ./CloudBeesFlow-<version>▼
```
Do one of the following to start the installation:
- For Linux with root or sudo privileges or for Windows installations, double-click the installer file.
- For Linux non-root/non-sudo installations, on a command line enter:
  ./CloudBeesFlow-<version> --nonRoot
  ▼
  For this installation type, a warning about automatic server start-up with non-root/non-sudo installations appears. Click Yes to dismiss the warning.
The install wizard welcome screen displays.
Select one of the following
- Install CloudBees Analytics prerequisites with built-in database: this option installs the built-in database, MariaDB, for use by the CloudBees Software Delivery Automation server.
- Install CloudBees Analytics prerequisites without built-in database: this option does not install a database. After installation and during CloudBees Software Delivery Automation server startup, you are asked to configure a database.
Select Next to continue. The Directories screen appears.
Review Install directory and Data directory default locations; select Browse to specify different directory locations. Click Next to continue. The Server service account screen appears.
Select the appropriate step for your platform and complete the information for the server service account.
- Windows:
  - User Name: Enter the name of the user who will run the CloudBees Software Delivery Automation server, web server, and repository server services.
  - Password: Enter the password of the user who will run the CloudBees Software Delivery Automation server, web server, and repository server services.
  - Domain: Enter the domain name information for the user. Leave this field blank if this is a local user.
  - Use the local system account: Select this check box if you want the CloudBees Software Delivery Automation server, repository server, and web server services to run as the local Windows system account.
  - Use the same account for the agent service: Select this check box if you want the agent on the CloudBees Software Delivery Automation server machine to run as the same account.
    
    For security reasons in production environments, CloudBees recommends using a separate account for the agent service because the server account has permission to read the key file located here:
- Linux: installation-directory/conf/passkey
- Windows: installation-directory\conf\passkey
  
  The key file is used to decrypt passwords stored in CloudBees Software Delivery Automation. Using a different account for the agent service ensures that a process running on the agent cannot gain access to the key file.
- Linux:
  - User Name: Enter the name of the user who owns the CloudBees Software Delivery Automation server, repository server, and web server processes.
  - Group Name: Enter the name of the group who owns the CloudBees Software Delivery Automation server, repository server, and web server processes.
  - Use the same account for the agent service: Select this check box if you want the agent on the CloudBees Software Delivery Automation server machine to run as the same account.
  Click Next to continue. The Agent Service Account screen appears.
If you selected the Use the same account for the agent service check box on the previous screen, you will not see this screen.
Select the appropriate step for your platform and complete the information for the agent service account.
- Windows:
  - User Name: Enter the name of the user who will run the CloudBees Software Delivery Automation agent service.
    
    The user that the agent runs as must have permission to write to the $INSTALL_DIRECTORY/log directory.
  - Password: Enter the password of the user who will run the CloudBees Software Delivery Automation agent service.
  - Domain: Enter the domain name information for the user. For example, electric-cloud.com. Leave this field blank if this is a local user.
  - Use the local system account: Select this check box if you want the CloudBees Software Delivery Automation agent service to run as the local Windows system account.
    
    The local system account does not have access to network shares.
- Linux:
  - User Name: Use this field to enter the name of the user who owns the CloudBees Software Delivery Automation agent process.
    
    The user/group that the agent runs as must have permission to write to the $INSTALL_DIRECTORY/log directory. If you specify root, click Yes when the following confirmation appears:
  - Group Name: Use this field to enter the name of the group that owns the CloudBees Software Delivery Automation agent process.
Click Next to continue. The Ready to Install screen appears.
Review the default settings and your service account selections. Use the Back button to change your service account selections if necessary.
Review Send user data information. When checked, anonymized, aggregated usage information is sent to CloudBees. Uncheck if you do not want to participate. Click Next to continue.

The installer displays a status bar to show the progress of the installation, which can take fifteen minutes. When the install process is complete, the Install Wizard Complete screen appears.

The CloudBees Software Delivery Automation server automatically starts when the installation is complete.
Select the Launch a web browser to login to CloudBees Software Delivery Automation check box if you want CloudBees Software Delivery Automation to open the login screen after installation is complete.
Click Finish to close the wizard.
For non-root/non-sudo Linux installations, configure autostart for the CloudBees Software Delivery Automation services.

For instructions, see Configuring autostart services for non-root/non-sudo Linux installations.

If you opted for installation without the built-in database, you are prompted on the sign-in page to configure an external database during CloudBees CD/RO server startup. See External database configuration for further information.

Installing CloudBees Analytics

The next step is to install CloudBees Analytics on a separate server.

Start the installation by following the steps for using the CloudBees Analytics UI installer.