CloudBees Pipeline Stage View Extensions causes Performance Issues when looking up for Checkpoints

Issue

Pipeline main page is slow to display

Thread dumps shows BLOCKED threads processing checkpoints / artifacts:

  "Handling GET /path/to/job/wfapi-checkpoints/ from 127.0.0.1 : Jetty (winstone)-921545" id=921545 (0xe0fc9) state=RUNNABLE cpu=44%
      at sun.nio.fs.UnixNativeDispatcher.realpath0(Native Method)
      at sun.nio.fs.UnixNativeDispatcher.realpath(UnixNativeDispatcher.java:262)
      at sun.nio.fs.UnixPath.toRealPath(UnixPath.java:834)
      at jenkins.util.VirtualFile$FileVF.isIllegalSymlink(VirtualFile.java:742)
      at jenkins.util.VirtualFile$FileVF.list(VirtualFile.java:597)
      at hudson.model.Run.addArtifacts(Run.java:1198)
      at hudson.model.Run.addArtifacts(Run.java:1223)
      at hudson.model.Run.addArtifacts(Run.java:1223)
      at hudson.model.Run.addArtifacts(Run.java:1223)
      at hudson.model.Run.addArtifacts(Run.java:1223)
      at hudson.model.Run.access$100(Run.java:153)
      at hudson.model.Run$AddArtifacts.call(Run.java:1190)
      at hudson.model.Run$AddArtifacts.call(Run.java:1179)
      at jenkins.util.VirtualFile.run(VirtualFile.java:496)
      at hudson.model.Run.getArtifactsUpTo(Run.java:1159)
      at com.cloudbees.workflow.rest.external.RunExt.createMinimal(RunExt.java:251)
      at com.cloudbees.workflow.rest.external.RunExt.createNew(RunExt.java:315)
      at com.cloudbees.workflow.rest.external.RunExt.create(RunExt.java:307)
      at com.cloudbees.workflow.flownode.FlowNodeUtil.getStageNodes(FlowNodeUtil.java:237)
      at com.cloudbees.workflow.pipeline.stageview.rest.CloudBeesFlowNodeUtil.getStageCheckpoints(CloudBeesFlowNodeUtil.java:45)
      at com.cloudbees.workflow.pipeline.stageview.rest.RunCheckpointAPI.getCheckpointInfo(RunCheckpointAPI.java:65)
      at com.cloudbees.workflow.pipeline.stageview.rest.JobCheckpointAPI.doDynamic(JobCheckpointAPI.java:49)
  [...]

Environment

Related Issue(s)

Pipeline Stage View #208

Explanation

The CloudBees Pipeline Stage View Extensions plugin adds a Retry icon on the stage view on stages that have checkpoints. See Inserting Checkpoints. This feature uses the workflow API to retrieve the checkpoints. It eventually looks up for artifacts of the pipeline build. Depending on the structure of the pipeline and the number of artifacts, this can be consuming.

Note that this check for checkpoints happens even when checkpoints are not used.

An improvement in Pipeline Stage View 2.23 was made to lookup for only the first artifact.

Resolution

First upgrade Pipeline Stage View plugin to version 2.23. It will be available under the CloudBees Assurance Program in the April 2022 release.

If the problem still occurs, disable the CloudBees Pipeline Stage View Extensions plugin. The only impact is the loss of the restart icon in the stage view UI. Checkpoints can still be access and actioned in the Checkpoints link in the sidebar of the build.

Workaround

First add the system property com.cloudbees.workflow.rest.external.RunExt.maxArtifactsCount=1 to limit the lookup to 1 artifact.

If the problem still occurs, disable the CloudBees Pipeline Stage View Extensions plugin. The only impact is the loss of the restart icon in the stage view UI. Checkpoints can still be access and actioned in the Checkpoints link in the sidebar of the build.