Issue
CloudBees CD is using Hikari CP to manage the server connection pools to the DB.
In some cases, Hikari CP component is not able to create new connections after some of them have exceeded maxlife time.
In this case, you will detect a big slowness in the CloudBees CD Server, with a long loading page time in the Web UI.
You can identify this scenario reviewing the commander.log
file, and looking for entries with the keyword connection has passed maxLifetime
similar to:
2021-05-12T04:47:54.055 | cdservername | DEBUG | qtp751321846-25102 | | | | ApiServletImpl | servletRequestContext[id=1058,on 16.192.120.52:8443,from 16.192.120.62:38196,bytes=133]: { "version" : "2.0", "sessionId" : "[PROTECTED]", "requests" : [ { "operation" : "getServerStatus", "parameters" : { "serverStateOnly" : true }, "requestId" : "24" } ] } ... 2021-05-12T05:11:40.738 | cdservername | DEBUG | HikariCP-001|||| HikariPool| CloudBees Flow - After cleanup stats (total=4, active=0, idle=4, waiting=0) 2021-05-12T05:12:00.426 | cdservername | DEBUG | HikariCP-4286|||| LogThreadCallback| Starting Thread[HikariCP-4286,5,main] 2021-05-12T05:12:00.426 | cdservername | DEBUG | HikariCP-4286|||| PoolBase| CloudBees Flow - Closing connection oracle.jdbc.driver.T4CConnection@3a4e8441: (connection has passed maxLifetime) 2021-05-12T05:12:00.427 | cdservername | DEBUG | HikariCP-4287|||| LogThreadCallback| Starting Thread[HikariCP-4287,5,main] 2021-05-12T05:12:00.480 | cdservername | DEBUG | HikariCP-4287|||| HikariPool| CloudBees Flow - Added connection oracle.jdbc.driver.T4CConnection@563a9ce0 2021-05-12T05:12:05.429 | cdservername | DEBUG | HikariCP-4286|||| LogThreadCallback| Exiting Thread[HikariCP-4286,5,main] 2021-05-12T05:12:05.429 | cdservername | INFO | HikariCP-4286||| ThreadStateCleanup| Contention: waitedCount=1 waitedTime=5000ms blockedCount=0 blockedTime=0ms .... 2021-05-12T05:17:54.056 | cdservername | DEBUG | qtp751321846-26163 | | | | JsonApiRequestHandler | Response for id=1058: { "responses" : [ { "error" : { "where" : "", "message" : "Request timed out", "details" : "", "code" : "RequestTimeout" } } ] }
Resolution
To solve this issue, we recommend you restart the CloudBees CD Server service, as explained in this article.
After the CloudBees CD Server service restarts the system should perform properly again.