Summary
You are running into issues where your Commander server is suddenly slowing down or some API requests taking way too long to complete. You need some help narrowing down what could be stalling this operation.
Solution
Thread dumps: These are sometimes printed into Commander server logs, but during an issue, its best to get a few of these thread dumps every 15 seconds to 1 minute.
These results will get printed into the commander-service.log and will help narrow down what threads are unavailable or stalled.
How to get Thread dumps
To get the thread dump as explained in https://wrapper.tanukisoftware.com/doc/english/integrate-start-stop-nix.html do steps below:
1- The ProcessID of the Commander wrapper process is stored in /opt/electriccloud/electriccommander/server/commander.pid file In this file if you see the content is "1052" that means the Commander wrapper process ID is 1052. 2- To get the thread dumps every 15 seconds you can run:
$ watch -n 15 "kill -3 < ProcessID of the Commander wrapper process>"
For example:
$ watch -n 15 "kill -3 1052"
Or
$ watch -n 15 "kill -3 `ps -ef | grep -ie "COMMANDER_HTTPS_PORT=8443" | grep -v grep | awk '{print $3}'`"
3- Once you get 5 minutes of thread dumps you can stop the watch command with a ctrl+c
4- Repeat these steps in all the CloudBees CD (CloudBees Flow) server Nodes in an HA environment.
Note:15 seconds is just an example. Depending on how long the slowness is or if the slowness is due to a burst of requests received by CloudBees CD (CloudBees Flow) server, this may have to be altered to 5 or 10 second intervals. Please follow the interval given by agent.
All the thread dumps will be logged to the /opt/electriccloud/electriccommander/logs/commander-service.log
Please send us:
-
Zip and send the commander-service.log file ( and if it rolled over then the recent server*.log.zip file which got rolled over).
-
Commander log files.