Jenkins Startup logs show builds migrated by the RunIdMigrator

Issue

Jenkins startup shows logs like the following:

jenkins.model.RunIdMigrator#migrate: Migrating build records in <pathToJobDirectory>/builds

Environment

Related Issue(s)

JENKINS-64356: legacyIds missing when creating jobs via REST API or CLI

Explanation

The RunIdMigrator is a component that guarantee compatibility for jobs that were created prior to version 1.597 of Jenkins by carrying out a "migration" of the suspected build records. The migration happens when jobs are loaded (usually on startup). It scan the ${JOB_DIRECTORY}/builds/ directory to check if builds need to be migrated to match the post 1.597 version structure.

It is based on a marker file ${JENKINS_HOME}/jobs/<pathToJob>/builds/legacyIds. If that file exists and is empty, then no migration is done. Otherwise, the job’s builds are migrated.

In case of false positives - for example if the legacyIds does not exist but the ${JOB_DIRECTORY}/builds/ directory structure is matching the post 1.597 version structure - the RunIdMigrator will not modifying existing files and just create the legacyIds that was missing. However it is still going through all the files and directories under the ${JOB_DIRECTORY}/builds/ that can cause some overhead. When creating jobs, Jenkins usually takes care of adding the empty legacyIds marker file. They are a few edge cases however when the legacyIds file is not created: when creating jobs using the REST API or the Jenkins CLI (see JENKINS-64356).

Resolution

If jenkins.model.RunIdMigrator#migrate: Migrating build records in <pathToJobDirectory>/builds is seen in startup logs, in most cases when the legacyIds file is missing and the root cause is related to the way jobs are created as in JENKINS-64356.

When creating jobs using the REST API, avoid creating directly from config.xml. Instead, create the empty job first and then update it with the config.xml:

# create a pipeline job from scratch
curl -X POST \
	 -u <user>:<pass> \
	 -H "Content-Type:application/x-www-form-urlencoded" \
	 "https://jenkins.example.com/createItem?name=TEST_JOB&mode=org.jenkinsci.plugins.workflow.job.WorkflowJob"

# update the job configuration by providing the XML
curl -X POST \
	 -u <user>:<pass> \
	 -H "Content-Type:application/xml" \
	 -d @config.xml \
	 "https://jenkins.example.com/job/TEST_JOB/config.xml"

the mode parameter of the /createItem API is the type of Job to create. You can find the available type with a groovy script like hudson.model.Items.LIST.each {println it.id}. Note that this issue only impacts buildable items such as Pipeline job or Freestyle jobs, but not Folders nor Pipeline Multibranch for example.

There is currently no solution when using the Jenkins CLI. See the workaround.

Workaround

Create the missing legacyIds manually before restarts. For example using bash:

#!/bin/bash

if [ ! -d "${JENKINS_HOME}/jobs" ]; then
    echo "JENKINS_HOME=${JENKINS_HOME} seems wrong"
fi

# Find all Job builds directory with a missing 'legacyIds' file.
comm -1 -3 --zero-terminated \
  <(find ${JENKINS_HOME}/jobs -depth -type f -iname 'legacyIds' -printf '%h\0' | sort -z) \
  <(find ${JENKINS_HOME}/jobs -depth -type d -iname 'builds' -prune -print0 | sort -z) | while read -d $'\0' file;
do
  echo "Migrating build records in ${file}"
  touch "${file}/legacyIds"
done