Introduction
IntroductionPre-installation requirementsInstallingVerifying Docker imagesUninstalling
IntroductionPre-installation requirementsInstallingVerifying Docker imagesUninstalling
IntroductionPre-installation requirementsInstallingVerifying Docker imagesUninstalling
IntroductionPre-installation requirementsInstallingVerifying Docker imagesUninstalling
IntroductionPre-installation requirementsInstallingVerifying Docker imagesUninstalling
IntroductionPre-installation requirementsInstallingVerifying Docker imagesUninstalling
IntroductionSystem requirementsVerifying Docker imagesInstalling operations centerInstalling client controllersInstalling build agentsVerifying build componentsVerifying WAR files
IntroductionHA (active/active) fundamentalsInstallation on modern cloud platforms (active/active)Installation on traditional platforms (active/active)HA (active/active) considerationsSpecific High Availability (active/passive) installation for CloudBees CI on traditional platforms
Introduction
IntroductionPipeline termsPipeline project typesCloudBees Pipeline benefits
IntroductionUsing Pipeline development utilitiesDetermining plugin compatibilityPipeline best practices
IntroductionPrerequisitesCreating your first PipelineSelecting agentsCreating Pipelines in SCMCreating Pipelines in Web UIUsing Pipeline as Code
IntroductionConfig Adv ScriptedConfig Build StageConfig Deploy StageConfig Optional Step ArgsConfig Test StageCreating JenkinsfileCustomizing ParametersHandling FailuresString InterpolationUsing Multiple AgentsWorking With The Env
IntroductionManaging artifactsTracing artifactsTriggering jobs with a simple webhookRestoring filesVisualizing PipelineInserting checkpointsSecuring PipelinesConfiguring Pipelines with user-scoped credentialsUsing Pipeline PoliciesSpecifying a matrix of one or more dimensionsConverting a Freestyle project to a Declarative PipelinePipeline builds and HA (active/active)
IntroductionRestarting aborted buildsLong-running buildsSkip next buildConsolidated Build View pluginCloudBees Quiet Start pluginCloudBees Template pluginCloudBees Git Validated Merge pluginCloudBees Workspace Caching
IntroductionTrigger a job with a notification event using Cross Team CollaborationEnable external notification events with external HTTP endpointsCluster-wide copy artifactsCluster-wide job triggers
IntroductionSetting up a Pipeline Template CatalogDefining Pipeline Template CatalogsSetting Up A Pipeline TemplateUsing parameters in template.yamlManaging multibranch Pipeline options in template.yamlManaging Pipeline Template Catalogs in bulkExample full Maven/Java app Jenkinsfile
IntroductionUsing Declarative Pipeline syntaxUsing Scripted Pipeline syntax
IntroductionBranch SourceBitbucketGitBranch Property StrategyExamples
CloudBees Docker Build and Publish pluginCloudBees Docker TraceabilityDocker Pipeline pluginCloudBees Docker Hub/Registry Notification plugin
Introduction
Introduction
IntroductionGetting startedNavigating the operations centerProvisioning managed controllers in multiple Kubernetes clustersProvisioning agents in a separate Kubernetes cluster from a managed controllerProvisioning a controller in a different namespace than the operations centerProvisioning a controller in a different OpenShift project than the operations centerManaging controllersManage controllers in specific Kubernetes namespacesMigrating an existing managed controller to High Availability (HA)Managing agentsShared agentsShared configurationShared cloud configurationTrigger restrictionsChanging NFS storage locationQuiet startMove/Copy/PromoteCluster operationsInbound agentsUsing KanikoUsing Buildkit (docker buildx)Sidecar injector for self-signed certificates on KubernetesSidecar injector for self-signed certificates on OpenShiftUsing auto-scaling nodes on EKSUsing auto-scaling nodes on GKECloudBees Amazon Web Services Deploy EngineAmazon Web Services CLI PluginCloud Foundry CLI PluginIntegrate OpenShift CLIServiceNow integrationCreating projects based on GitHub repository structureUsing GitHub App authenticationWikiText plugin
IntroductionGetting startedNavigating the Operations CenterManaging client controllersMigrating from High Availability (active/passive) to High Availability (active/active) on CloudBees CI on traditional platformsManaging agentsShared agentsShared configurationShared cloud configurationTrigger restrictionsQuiet startMove/Copy/PromoteCluster operationsJava web start agentsServiceNow integrationCreating projects based on GitHub repository structureUsing GitHub App authenticationWikiText plugin
Introduction
IntroductionTrust modelPod Security AdmissionCentrally managing security for controllersCyberark Credential Provider PluginHashiCorp Vault PluginEnabling advanced use cases: cross controller triggers and bulk operationsRestricting access and delegating administration with Role-Based Access ControlCreating secure folders with Role-Based Access Control Auto ConfigurerRestricting job triggersSetting up operations center access controlsSetting up access controls on connected controllersTesting the SSH connection to an agentConfiguring restricted credentials with the CloudBees Restricted Credentials pluginFoldersFolders PlusInjecting secrets into buildsManaging build agents with Nodes Plus pluginExtended security settingsEnhanced credentials maskingUnderstanding Beekeeper security warningsPreventing unauthorized interaction between buildsSecurity recommendationsUsing single sign-on (SSO)CloudBees CI integration with Microsoft Entra IDoperations center specific permissionsAuthentication mappingExample configurationsDelegating administrationData collection for the CloudBees Analytics PluginExternal secrets managementReplacing an expired certificateList of URLs that need accessBlocking access to URL patterns using the CloudBees Request Filter PluginServing resources from JenkinsVerifying Helm Charts Published with a Signature
IntroductionConfiguring network requirementsCentrally managing security for controllersCyberark Credential Provider PluginHashiCorp Vault PluginEnabling advanced use cases: cross-controller triggers and bulk operationsRestricting access and delegating administration with Role-Based Access ControlCreating secure folders with Role-Based Access Control Auto ConfigurerRestricting job triggersTesting the SSH connection to an agentConfiguring restricted credentials with the CloudBees Restricted Credentials pluginFoldersFolders PlusInjecting secrets into buildsManaging build agents with Nodes Plus pluginUnderstanding Beekeeper security warningsPreventing unauthorized interaction between buildsSecurity recommendationsExtended security settingsUsing single sign-on (SSO)CloudBees CI integration with Microsoft Entra IDOperations Center specific permissionsAuthentication mappingExample configurationsDelegating administrationData collection for the CloudBees Analytics PluginExternal secrets managementList of URLs that need accessBlocking access to URL patterns using the CloudBees Request Filter PluginServing resources from Jenkins
Introduction
IntroductionExplaining $JENKINS_HOMEBest practices for backup and restoreBacking up manuallyRestoring manuallyRestoring credentialsUsing the CloudBees Backup PluginScheduling your backups in the CloudBees Backup PluginRestoring from CloudBees Backup PluginBackup and restore on KubernetesBackup and restore on AWSUsing Velero for backup and restore of Kubernetes clustersUsing cluster operations to run backups
IntroductionGetting the CLI toolUsing the CLI toolExamples of usageConfiguring multiple client controllersConfiguring an alias for the Jenkins CLI toolConfiguring the Jenkins CLI tool to work with non-TrustStore SSL certificatesCasC bundle management on operations center
CloudBees Inactive Items Plugin
Jenkins Health Advisor by CloudBees
CloudBees Pull Request Builder for GitHub plugin
Count and monitor user licenses with the CloudBees User Activity Monitoring plugin
Introduction
CasC fundamentals
CasC requirements
CasC permissions
IntroductionGetting startedCreating a bundleConfiguring the operations center on modern platformsConfiguring the operations center on traditional platformsRetrieving bundles using SCMUpdating a bundleBundle update timing for the operations centerViewing the operations center update logCreating itemsConfiguring RBACDetermining plugin compatibilityTroubleshooting
IntroductionGetting startedCreating a bundleAdding controller bundles to the operations centerUpdating a bundleBundle update timing for controllersViewing the controller update logConfiguring bundle availabilitySetting up a managed controllerSetting up a client controllerCreating itemsConfiguring RBACDetermining plugin compatibilityAdvanced topicsTroubleshooting
Plugin management with CasC
CasC CLI commands
CasC HTTP API
Support policies

High Availability (active/active) troubleshooting

4 minute read

If CloudBees High Availability (HA) is not working as expected, use this page to troubleshoot.

High Availability (HA) developer mode

The High Availability (HA) feature provides a developer mode to troubleshoot HA problems in controllers running in HA mode.

To enable the developer mode in a controller.

  • Navigate to Manage Jenkins  CloudBees CI High Availability.

  • Select Status on the left navigation pane. This is the default view when accessing the CloudBees CI High Availability screen.

  • Select the Enable developer mode field and select Save or Apply.

Enable developer mode
Figure 1. Enable developer mode

Developer mode is a powerful tool to troubleshoot and understand High Availability. When enabled, the controller provides additional information about the High Availability mode:

  • A button with the current replica name appears in the page footer.

  • The background color for the replica button changes from one replica to another.

  • When selected, the replica button on the footer redirects to the CloudBees CI High Availability screen.

    Footer in developer mode
    Figure 2. Footer in developer mode.

  • The consolidated queue widget displays the queue items in all the replicas.

  • In developer mode, CloudBees CI adds the replica name to those items queued in other replicas.

  • Names for items queued in the current replica remain the same.

Build name in developer mode
Figure 3. Build name in developer mode

In addition to enabling developer mode, also from the CloudBees CI High Availability screen, users can change the replica they are using by selecting the Reset sticky session button.

Switch to another replica
Figure 4. Switch to another replica

When the Reset sticky session is selected, CloudBees CI randomly assigns a new replica. If there is no change and you are assigned to the same replica, you can reload the page and try again until a new replica is assigned.

The Reset sticky session button only displays if you are using ingress-nginx. However, you may be assigned to a different replicas if, while using your browser developer tools, you remove CloudBees CI cookies and sign in again.

High Availability (HA) Script Console

The CloudBees CI High Availability screen provides a High Availability Script Console. This console allows CloudBees CI users to run scripts across all the current controller replicas and displays the results. To access the HA Script Console, select Script Console on the left.

HA Script Console
Figure 5. HA Script Console

  1. Select Script Console to access the HA Script Console.

  2. Type your scripts in the scripts area.

  3. Run your HA scripts. When selected, the script will be executed in all the controller replicas.

Results for the HA Script Console
Figure 6. Results for the HA Script Console

Include HA information in your support bundle

The CloudBees Support Plugin allows you to generate a support bundle that contains commonly requested diagnostic information used by CloudBees to resolve support issues.

This plugin is installed by default with CloudBees CI. For more information about how to generate support bundles and the information collected by CloudBees from support bundles, refer to Generating a support bundle.

To include specific High Availability (HA) information in a controller support bundle:

  1. From the root of you controller, select Support on the left navigation pane.

  2. Select the Information from other replicas option in the CloudBees Support screen. When this option is selected, the generated support bundle contains the HA information in the replicas/ directory. The replicas/ directory will include the following subdirectories:

    • The live/ directory contains a subdirectory with data for each current replica.

    • The exited/ directory contains contains a subdirectory with data for each replica previously used and replaced. For example, replicas that were replaced during rolling restarts or rolling upgrades.

  3. Select Generate Bundle to generate and download the support bundle with the HA information.

Include specific HA information in your controller support bundle
Figure 7. Include specific HA information in your controller support bundle
This option is only available for controllers that run in HA mode.

HTTP Proxy Configuration should include replicas

If configuring a proxy for your CloudBees CI installation, ensure that the proxy configuration includes all the replicas' IPs and hostnames in the No Proxy Host field. This is important because the replicas should communicate with each other directly.

Troubleshoot CloudBees CI on modern cloud platforms installations

Controller configuration is not set to Deployment

If the controller fails to provision with the error:

ERROR: Failed to provision controller ... StatefulSet is only for non-replicated controllers

The issue was caused by including kind: StatefulSet when configuring the controller under Advanced configuration → YAML. To resolve the issue, select Acknowledge error, Free snapshot, then go to the configuration page and change kind: StatefulSet to kind: Deployment in the YAML field.

Troubleshoot CloudBees CI on traditional platforms installations

Replicas not forming a controller cluster

If a controller replica is not forming the cluster, it will not be displayed in the CloudBees CI High Availability screen, and the logs may contain an error similar to:

[cloudbees-replication] [5.3.6] [IP]:5701 is added to the blacklist.
...
INFO	c.h.internal.cluster.ClusterService: [IP]:5701 [cloudbees-replication] [5.3.6]

Members {size:1, ver:1} [
	Member [IP]:5701 - ID this
]
...
[OperationsCenter2 connection to HOSTNAME/IP:50000] Local headers refused by remote: The controller 0-client-controller is already connected
▼

These troubleshooting steps may help solve the issue:

  1. Review the installation steps and the required JVM arguments as explained in the Installation for CloudBees CI on traditional platforms (active/active) page.

  2. Log in to one of the replicas, and check the IP and port listed in the directory: JENKINS_HOME/cloudbees-replication-discovery/*. This directory should contain one file per replica.

  1. If the replica IP and port are what you expect, the problem is likely a network configuration problem. You must check if the replicas can communicate with each other over the IP and port from the JENKINS_HOME/cloudbees-replication-discovery/* directory.

  2. If any IP or port has an unexpected value, Hazelcast likely chose another network interface on the machine to bind to. You can configure Hazelcast to bind to a specific network interface by adding the following additional JVM arguments to each replica.

-Dhz.network.interfaces.enabled=true \
-Dhz.network.interfaces.interfaces.interface1='<IP_PATTERN>' \ (1)
-Dhz.network.port.port=5701 \
-Dhz.network.port.autoincrement=true \
-Dhazelcast.jmx=false \
-Dhazelcast.metrics.jmx.enabled=false \
-Dhazelcast.health.monitoring.delay.seconds=180 \
-Dhazelcast.health.monitoring.threshold.memory.percentage=99 \
-Dhazelcast.health.monitoring.threshold.cpu.percentage=99 \
▼
1 Replace <IP_PATTERN> with the IP pattern of the desired network interface. For example, 10.3.10.* searches for a network interface on the machine with an IP between 10.3.10.0 and 10.3.10.255.
Enable the options above on all replicas, and restart them after the change.

  1. If the issue remains unsolved, enable the Hazelcast diagnostic flag in your JVM startup argument, -Dhazelcast.diagnostics.enabled=true, and add a custom logger to the class com.cloudbees.jenkins.plugins.replication.hazelcast.FilesystemDiscoveryStrategy with the level FINE to get more information about the replica discovery process.

High Availability (active/passive) installation troubleshooting
Collecting cluster logs