How to Troubleshoot the Root Cause of RunIdMigrator occurrences

Article ID:360052964732
1 minute readKnowledge base

Symptoms

  • Jenkins takes a long time to start

  • Jenkins shows the following logs on start up:

jenkins.model.RunIdMigrator#migrate: Migrating build records in <pathToJobDirectory>/builds

Diagnostic/Treatment

The following recommendation and techniques help to troubleshoot this issue and identify the root cause. They are based on experience with CloudBees customers.

Preconditions

This problem is caused in most cases by the creation of Jobs using the Jenkins REST API / Jenkins CLI.

Before troubleshooting any further, we recommend to look at the following article Jenkins Startup logs show builds migrated by the RunIdMigrator.

Troubleshooting

The approach is to:

1) Find what jobs that were impacted in previous startups and what jobs will be impacted in next startups by checking the (in)existence of the legacyIds files. 2) Check how these jobs are created 3) Fix the creation script using the solution / workaround presented in Jenkins Startup logs show builds migrated by the RunIdMigrator.

Identify Jobs Migrated in previous startup

Find all jobs that were migrated on the previous startups by looking at the Jenkins logs:

grep "jenkins.model.RunIdMigrator#migrate: Migrating" jenkins.log | sed 's/.*Migrating build records in \(.*\)\/builds/\1/' | sort

Then check how those jobs were created.

Identify Jobs currently Impacted

Find all jobs directories that miss a legacyIds file:

comm -1 -3 \
    <(find ${JENKINS_HOME}/jobs -depth -type f -iname 'legacyIds' -printf '%h\n' | sort) \
    <(find ${JENKINS_HOME}/jobs -depth -type d -iname 'builds' -prune -print | sort)

Find all jobs directories that have a legacyIds file that is not empty:

find ${JENKINS_HOME}/jobs -depth -type f -iname 'legacyIds' -size +0 -print

Then check how those jobs were created.

Data Collection

In order to troubleshoot the issue further:

  • Collect a Support Bundle

  • Collect the Jenkins logs since the beginning of the startup

  • Collect the list of jobs with missing legacyIds using:

      comm -1 -3 \
  <(find ${JENKINS_HOME}/jobs -depth -type f -iname 'legacyIds' -printf '%h\n' | sort) \
  <(find ${JENKINS_HOME}/jobs -depth -type d -iname 'builds' -prune -print | sort) > without-legacyIds.log

  • Collect the list of jobs with legacyIds using:

      find ${JENKINS_HOME}/jobs -depth -type f -iname 'legacyIds' -print | sort > with-legacyIds.log