SlaveLeaseTable H2 DB exception after core upgrade

2 minute readKnowledge base

Issue

  • After upgrading/migrating my CloudBees CI cluster to 2.332.3.4 or later I am unable to provision new managed controllers. In the operations center logs I observe the following:

SEVERE c.g.c.u.c.AbstractFuture#executeListener: RuntimeException while executing runnable CallbackListener{com.cloudbees.opscenter.server.model.ManagedMaster$ProvisioningHandler@20bb6273} with executor com.google.common.util.concurrent.MoreExecutors$DirectExecutorService@4acf0765
java.lang.IllegalStateException: Could not create grant
▼
WARNING c.c.o.s.p.SlaveLeaseTable#getLeases: Failed to get leases in state: REQUESTED
org.h2.jdbc.JdbcSQLNonTransientException: General error: "The write format 1 is smaller than the supported format 2 [2.1.210/5]" [50000-210]
[...]
Caused by: org.h2.mvstore.MVStoreException: The write format 1 is smaller than the supported format 2 [2.1.210/5]
[...]
▼

  • After upgrading/migrating my CloudBees CJP cluster to 2.332.3.4 or later I am unable to edit any shared agents information. In the operations center logs I observe the following message:

WARNING [com.cloudbees.opscenter.server.model.SharedSlave.PeriodicTidyUp thread] com.cloudbees.opscenter.server.persistence.SlaveLeaseTable.getLeaseRecords Failed to get lease records for owner: ff66b2d6-2957-44fa-bce2-7f0ce72ffb51
        org.h2.jdbc.JdbcSQLNonTransientException: General error: "The write format 1 is smaller than the supported format 2 [2.1.210/5]"
[...]
        Caused by: org.h2.mvstore.MVStoreException: The write format 1 is smaller than the supported format 2 [2.1.210/5]
[...]
▼

  • After upgrading/migrating my CloudBees cluster to 2.332.3.4 or later, in the operations center logs I observe similar warnings similar to:

org.h2.jdbc.JdbcSQLNonTransientException: General error: "The write format 1 is smaller than the supported format 2 [2.1.210/5]" [50000-210]
[...]
Caused by: org.h2.mvstore.MVStoreException: The write format 1 is smaller than the supported format 2 [2.1.210/5]
[...]
▼

Environment

Context

Operations Center Server 2.332.0.4: Upgrade of the H2 database (BEE-16032) and Release notes 2.332.3.2 - Resolved Issues: Response to security scans regarding the H2 database (BEE-14368, BEE-14366, BEE-10343, BEE-14611, BEE-16032).

Routine security scans detected vulnerabilities in the 1.x version of the H2 database that was embedded in the operations center and in the User Activity Monitoring plugin. Although the H2 security vulnerabilities do not affect CloudBees CI, the H2 database was upgraded to version 2.x in the operations center as a best practice.

We have observed that, under certain circumstances, when having User Activity Monitoring plugin installed in an Operations Center instance, the markerfile used to store the status of the Shared Agents lease database migration, .shared_cloud_and_agent_lease_database_upgraded, gets corrupted causing problems to edit its content.

Workaround

  1. Remove the existing marker file $JENKINS_HOME/.shared_cloud_and_agent_lease_database_upgraded.

  2. Restart the Operations Center.

  3. Check if the problem is solved.

If the issue persists, open a ticket to CloudBees Support explaining the situation, indicating the steps that have been followed and attaching a Support Bundle of the affected instances.

Tested product environment