Issue
-
After upgrading CloudBees CI Controller to version 2.414.1.4, Blue Ocean and Pipeline views pages are slow to render and causing high CPU load. In some environment the controller may become unresponsive or restart unexpectedly.
-
Thread dumps show a lot of threads BLOCKED on
at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.findClosestEnclosingTreeNode(FlowNodeInfoUtils.java:269)Pipeline Stage View / Checkpoints:[...] at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.computeFlowGraph(FlowNodeInfoUtils.java:200) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.findClosestEnclosingTreeNode(FlowNodeInfoUtils.java:269) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.viewer.flownode.AdvancedConsoleTransientActionFactoryFlowNode.createFor(AdvancedConsoleTransientActionFactoryFlowNode.java:35) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.viewer.flownode.AdvancedConsoleTransientActionFactoryFlowNode.createFor(AdvancedConsoleTransientActionFactoryFlowNode.java:19) at hudson.model.Actionable.createFor(Actionable.java:115) at hudson.model.Actionable.getAllActions(Actionable.java:101) at org.jenkinsci.plugins.workflow.graph.FlowNode.getMaybeTransientAction(FlowNode.java:385) at org.jenkinsci.plugins.workflow.graph.FlowNode.getAction(FlowNode.java:402) at com.cloudbees.workflow.cps.checkpoint.CheckpointNodeAction.getAction(CheckpointNodeAction.java:59) at com.cloudbees.workflow.pipeline.stageview.rest.CloudBeesFlowNodeUtil.getStageCheckpoints(CloudBeesFlowNodeUtil.java:48) at com.cloudbees.workflow.pipeline.stageview.rest.RunCheckpointAPI.getCheckpointInfo(RunCheckpointAPI.java:65) at com.cloudbees.workflow.pipeline.stageview.rest.JobCheckpointAPI.doDynamic(JobCheckpointAPI.java:49) [...]
Blue Ocean (occurs when users load a Pipeline in Blue Ocean)
[...] at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.computeFlowGraph(FlowNodeInfoUtils.java:200) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.findClosestEnclosingTreeNode(FlowNodeInfoUtils.java:269) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.viewer.flownode.AdvancedConsoleTransientActionFactoryFlowNode.createFor(AdvancedConsoleTransientActionFactoryFlowNode.java:35) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.viewer.flownode.AdvancedConsoleTransientActionFactoryFlowNode.createFor(AdvancedConsoleTransientActionFactoryFlowNode.java:19) at hudson.model.Actionable.createFor(Actionable.java:115) at hudson.model.Actionable.getActions(Actionable.java:140) at io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeGraphVisitor.accumulatePipelineActions(PipelineNodeGraphVisitor.java:559) at io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeGraphVisitor.atomNode(PipelineNodeGraphVisitor.java:531) at org.jenkinsci.plugins.workflow.graphanalysis.ForkScanner.fireVisitChunkCallbacks(ForkScanner.java:765) at org.jenkinsci.plugins.workflow.graphanalysis.ForkScanner.visitSimpleChunks(ForkScanner.java:783) at org.jenkinsci.plugins.workflow.graphanalysis.ForkScanner.visitSimpleChunks(ForkScanner.java:647) at io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeGraphVisitor.<init>(PipelineNodeGraphVisitor.java:117) at io.jenkins.blueocean.rest.impl.pipeline.NodeGraphBuilder$NodeGraphBuilderFactory.getInstance(NodeGraphBuilder.java:39) at io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeContainerImpl.<init>(PipelineNodeContainerImpl.java:32) at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getNodes(PipelineRunImpl.java:190) [...]
GitHub Checks (occurs in background)
[...] at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.computeFlowGraph(FlowNodeInfoUtils.java:200) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.api.FlowNodeInfoUtils.findClosestEnclosingTreeNode(FlowNodeInfoUtils.java:269) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.viewer.flownode.AdvancedConsoleTransientActionFactoryFlowNode.createFor(AdvancedConsoleTransientActionFactoryFlowNode.java:35) at com.cloudbees.jenkins.plugins.pipeline.explorer.log.viewer.flownode.AdvancedConsoleTransientActionFactoryFlowNode.createFor(AdvancedConsoleTransientActionFactoryFlowNode.java:19) at hudson.model.Actionable.createFor(Actionable.java:115) at hudson.model.Actionable.getAllActions(Actionable.java:101) at org.jenkinsci.plugins.workflow.graph.FlowNode.getMaybeTransientAction(FlowNode.java:385) at org.jenkinsci.plugins.workflow.graph.FlowNode.getAction(FlowNode.java:402) at org.jenkinsci.plugins.workflow.support.visualization.table.FlowGraphTable$Row.<init>(FlowGraphTable.java:265) at org.jenkinsci.plugins.workflow.support.visualization.table.FlowGraphTable.createAllRows(FlowGraphTable.java:85) at org.jenkinsci.plugins.workflow.support.visualization.table.FlowGraphTable.build(FlowGraphTable.java:62) at io.jenkins.plugins.checks.status.FlowExecutionAnalyzer.extractOutput(FlowExecutionAnalyzer.java:150) at io.jenkins.plugins.checks.status.BuildStatusChecksPublisher.getOutput(BuildStatusChecksPublisher.java:118) at io.jenkins.plugins.checks.status.BuildStatusChecksPublisher$ChecksGraphListener.lambda$onNewHead$0(BuildStatusChecksPublisher.java:245) at io.jenkins.plugins.checks.status.BuildStatusChecksPublisher$ChecksGraphListener$$Lambda$2139/0x0000000842f93040.accept(Unknown Source) at java.base@11.0.20/java.util.Optional.ifPresent(Optional.java:183) at io.jenkins.plugins.checks.status.BuildStatusChecksPublisher$ChecksGraphListener.onNewHead(BuildStatusChecksPublisher.java:244) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1530) [...]
Environment
-
CloudBees CI on modern cloud platforms - operations center 2.414.1.4
-
CloudBees CI on modern cloud platforms - managed controller 2.414.1.4
-
CloudBees CI on traditional platforms - operations center 2.414.1.4
-
CloudBees CI on traditional platforms - client controller 2.414.1.4
-
CloudBees Pipeline Explorer plugin >= 1.3 and < 1.4.1
Related Issue(s)
-
BEE-40006: CloudBees Pipeline Explorer makes flow graph traversal extremely slow when FlowNode actions are loaded
Explanation
Since its version 1.3, the CloudBees Pipeline Explorer plugin performed an expensive computation which caused pages and APIs that load transient actions for multiple FlowNodes to be extremely slow for complex Pipelines, sometimes timing out. This affects the “Pipeline Steps“ view, Blue Ocean, and Pipeline Stage view when users interacts with those pages. When using github-checks, it affects overall build performance and Jenkins controller CPU load even without user interaction. For the triggers that require user interaction, enough users using those pages can affect overall controller performance.
There are likely other plugins that trigger the same issues, but the cases listed above are the known ones at this time.