Issue
CloudBees CD is using Hikari CP to manage the server connection pools to the DB.
In some cases, Hikari CP component is not able to create new connections after some of them have exceeded maxlife time.
In this case, you will detect a big slowness in the CloudBees CD Server, with a long loading page time in the Web UI.
You can identify this scenario reviewing the commander.log file, and looking for entries with the keyword connection has passed maxLifetime similar to:
2021-05-12T04:47:54.055 | cdservername | DEBUG | qtp751321846-25102 | | | | ApiServletImpl | servletRequestContext[id=1058,on 16.192.120.52:8443,from 16.192.120.62:38196,bytes=133]:
{
"version" : "2.0",
"sessionId" : "[PROTECTED]",
"requests" : [ {
"operation" : "getServerStatus",
"parameters" : {
"serverStateOnly" : true
},
"requestId" : "24"
} ]
}
...
2021-05-12T05:11:40.738 | cdservername | DEBUG | HikariCP-001|||| HikariPool| CloudBees Flow - After cleanup stats (total=4, active=0, idle=4, waiting=0)
2021-05-12T05:12:00.426 | cdservername | DEBUG | HikariCP-4286|||| LogThreadCallback| Starting Thread[HikariCP-4286,5,main]
2021-05-12T05:12:00.426 | cdservername | DEBUG | HikariCP-4286|||| PoolBase| CloudBees Flow - Closing connection oracle.jdbc.driver.T4CConnection@3a4e8441: (connection has passed maxLifetime)
2021-05-12T05:12:00.427 | cdservername | DEBUG | HikariCP-4287|||| LogThreadCallback| Starting Thread[HikariCP-4287,5,main]
2021-05-12T05:12:00.480 | cdservername | DEBUG | HikariCP-4287|||| HikariPool| CloudBees Flow - Added connection oracle.jdbc.driver.T4CConnection@563a9ce0
2021-05-12T05:12:05.429 | cdservername | DEBUG | HikariCP-4286|||| LogThreadCallback| Exiting Thread[HikariCP-4286,5,main]
2021-05-12T05:12:05.429 | cdservername | INFO | HikariCP-4286||| ThreadStateCleanup| Contention: waitedCount=1 waitedTime=5000ms blockedCount=0 blockedTime=0ms
....
2021-05-12T05:17:54.056 | cdservername | DEBUG | qtp751321846-26163 | | | | JsonApiRequestHandler | Response for id=1058:
{
"responses" : [ {
"error" : {
"where" : "",
"message" : "Request timed out",
"details" : "",
"code" : "RequestTimeout"
}
} ]
}
Resolution
To solve this issue, we recommend you restart the CloudBees CD Server service, as explained in this article.
After the CloudBees CD Server service restarts the system should perform properly again.
This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.