Client controller fails to start or to lease a Shared Agent because of org.mapdb.AsyncWriteEngine.checkState

Issue

The issue can manifest in at least two different ways:

Client controller fails to start, or
Client controller fails to lease shared agent from Operations Center

In both cases, the messages with a stack trace similar to the following should be present in the Client controller logs:

2016-01-06 00:00:56.688-0600 [id=96366] WARNING c.c.o.s.m.ConnectedMaster$RemoteSlaveManifest#getSlaves: Could not complete remote call
java.lang.RuntimeException: Writer thread failed
    at org.mapdb.AsyncWriteEngine.checkState(AsyncWriteEngine.java:262)
    at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:326)
    at org.mapdb.Caches$HashTable.get(Caches.java:245)
    at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1081)
    at org.mapdb.HTreeMap$HashIterator.findNextLinkedNode(HTreeMap.java:1046)
    at org.mapdb.HTreeMap$HashIterator.advance(HTreeMap.java:1033)
    at org.mapdb.HTreeMap$HashIterator.moveToNext(HTreeMap.java:992)
    at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1139)
    at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1132)
    at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.getSlaves(PluginImpl.java:422)
    at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2086)
    at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2083)
    at hudson.remoting.UserRequest.perform(UserRequest.java:120)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
    at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to 1-jenkins2 at https://<JENKINS_URL>.com/(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
    at hudson.remoting.Channel.call(Channel.java:781)
    at com.cloudbees.opscenter.server.model.ConnectedMaster$RemoteSlaveManifest.getSlaves(ConnectedMaster.java:2051)
    at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:865)
    at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:858)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 404
    at org.mapdb.StoreWAL.longStackPutSixLong(StoreWAL.java:1060)
    at org.mapdb.StoreWAL.longStackPut(StoreWAL.java:1040)
    at org.mapdb.StoreDirect.freePhysPut(StoreDirect.java:1055)
    at org.mapdb.StoreDirect.freePhysTake(StoreDirect.java:1081)
    at org.mapdb.StoreDirect.physAllocate(StoreDirect.java:655)
    at org.mapdb.StoreWAL.update(StoreWAL.java:436)
    at org.mapdb.EngineWrapper.update(EngineWrapper.java:63)
    at org.mapdb.AsyncWriteEngine.access$101(AsyncWriteEngine.java:76)
    at org.mapdb.AsyncWriteEngine.runWriter(AsyncWriteEngine.java:220)
    at org.mapdb.AsyncWriteEngine$WriterRunnable.run(AsyncWriteEngine.java:172)
    ... 1 more

2018-05-13 02:21:10.115+0000 [id=45]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed PluginImpl.afterExtensionsAugmented
java.lang.AssertionError: data were not fully read, check your serializer
    at org.mapdb.Store.deserialize(Store.java:299)
    at org.mapdb.StoreDirect.get2(StoreDirect.java:486)
    at org.mapdb.StoreWAL.get2(StoreWAL.java:336)
    at org.mapdb.StoreWAL.get(StoreWAL.java:320)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:333)
    at org.mapdb.Caches$HashTable.get(Caches.java:246)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.HTreeMap.getInner(HTreeMap.java:448)
    at org.mapdb.HTreeMap.getPeek(HTreeMap.java:436)
    at org.mapdb.HTreeMap.containsKey(HTreeMap.java:300)
    at org.mapdb.HTreeMap$KeySet.contains(HTreeMap.java:817)
    at java.util.AbstractCollection.retainAll(AbstractCollection.java:409)
    at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.createDb(PluginImpl.java:419)
    at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.<init>(PluginImpl.java:406)
    at com.cloudbees.opscenter.client.cloud.PluginImpl.getSlaveManifest(PluginImpl.java:310)
    at com.cloudbees.opscenter.client.cloud.PluginImpl.afterExtensionsAugmented(PluginImpl.java:330)
Caused: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
    at jenkins.model.Jenkins$7.runTask(Jenkins.java:1073)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2018-05-13 02:21:10.120+0000 [id=24]    SEVERE  hudson.util.BootFailure#publish: Failed to initialize Jenkins
java.lang.AssertionError: data were not fully read, check your serializer
    at org.mapdb.Store.deserialize(Store.java:299)
    at org.mapdb.StoreDirect.get2(StoreDirect.java:486)
    at org.mapdb.StoreWAL.get2(StoreWAL.java:336)
    at org.mapdb.StoreWAL.get(StoreWAL.java:320)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:333)
    at org.mapdb.Caches$HashTable.get(Caches.java:246)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.HTreeMap.getInner(HTreeMap.java:448)
    at org.mapdb.HTreeMap.getPeek(HTreeMap.java:436)
    at org.mapdb.HTreeMap.containsKey(HTreeMap.java:300)
    at org.mapdb.HTreeMap$KeySet.contains(HTreeMap.java:817)
    at java.util.AbstractCollection.retainAll(AbstractCollection.java:409)
    at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.createDb(PluginImpl.java:419)
    at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.<init>(PluginImpl.java:406)
    at com.cloudbees.opscenter.client.cloud.PluginImpl.getSlaveManifest(PluginImpl.java:310)
    at com.cloudbees.opscenter.client.cloud.PluginImpl.afterExtensionsAugmented(PluginImpl.java:330)
Caused: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
    at jenkins.model.Jenkins$7.runTask(Jenkins.java:1073)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1107)
    at jenkins.model.Jenkins.<init>(Jenkins.java:912)
    at hudson.model.Hudson.<init>(Hudson.java:86)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.WebAppMain$3.run(WebAppMain.java:235)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:252)

Environment

Operations Center Context Plugin

Resolution

mapdb database has become corrupted on the Jenkins client controller.

Stop the client controller
Delete the following files
- ${JENKINS_HOME}/operations-center-cloud.db
- ${JENKINS_HOME}/operations-center-cloud.db.p
- ${JENKINS_HOME}/operations-center-cloud.db.t
Restart the client controller to see if the problem is resolved:

If the problem persists after deleting those files, the messaging databases might be corrupted too. Please review this article HA Cluster does not start when messaging database is corrupted for its resolution.