Issue
The ectool getServerStatus --diagnostics 1 output contains lots of information - how can I use this?
Environment
-
CloudBees CD (CloudBees Flow) versions 10.3 and below.
Solution
The API for getServerStatus has 2 purposes:
A) Simple Server Status Check:
If you run getServerStatus directly from a machine that has the CD server software installed, and you have not logged in using ectool login <username> <password>, you will receive a simple output status with either value of 1 (up and running) or 0 (not up).
B) Deliver Diagnostic information:
Here is a summary of the sections of output provided and how to read this information. All data presented here is either fixed data, based on the current snapshot in time, or collective data since the last time this CD server was restarted.
1) ACL Statistics
-
a brief set of data concerning caching of ACLs
-
This is not likely to be relevant in most cases
2) Agents
-
This section provides information about recently used agents (cachedAgents) and their current status
-
This section may be useful in rare case when trying to understand the current status of an agent inside the system
3) apiMonitor
-
This section delivers information about the longest Call, active API calls, and recent API calls
-
This is not likely to be relevant in most cases, but could be used when hunting down jobs that appear to be stuck
4) clusterManager
-
This section provides data on Active MQ content
-
This is not likely to be relevant in most cases
5) clusterSingletonServices
-
This information identiies which server nodes are running which of the singleton services employed by the CD service. On a single node system, these will all be on the 1 node, but in an HA or multi-node CD system, these services are distributed to running nodes to help balance system workload.
-
This information can be useful for a user in identifying which node is running a service, like the backgroundDeleter or the step scheduler.
6) entityCounts
-
This section lists each table used in the CD database schema and the number of elements stored therein.
eg: This block of data shows that there are over 0.5M jobs in the current database:
<entityCount>
<name>job</name>
<value>523819</value>
</entityCount>
-
These metrics can be useful to users in understanding the overall scope of content being stored in the system, or when used as a comparison with past collections, in understanding the extent to which areas of the databas may be growing. Such questions may come into play when performance changes are being observed over time. Reviewing this data can help to recognize how trimming certain data types may provide some performance improvements.
-
When viewing this data, relative sizing can be more important than the sheer number.
eg: 1M properties may not bring any concern, as each property is essentially a leaf node, whereas 10K pipelines might be encouraging some performance impacts in viewing pipelines in some environments, since a pipeline contains numerous children objects (stages, tasks, properties, parameters, etc…).
Some of the more common elements to take note of would be:
1) entityChange - the # of items stored for the Change Tracking feature 2) flowRuntime - the # of pipeline runs (re - vs "pipeline" which shows the # of pipeline definitions) 3) job -the # of jobs 4) jobStep - the # of jobSteps across all jobs 5) property - the # of properties stored 6) propertySheet - the # of propertySheets or folders that are collecting other children properties - comparing the ratio against the property count can be informative in terms of how teams are storing data 7) resource - the # of defined resources currently in the database
7) entityLocks
-
This data is typically more of interest to the CloudBees engineering team
8) environment
-
This section lists relevant system environment variables picked up at the time the server was started
9) lazywiredPostProcessor
-
This data is typically more of interest to the CloudBees engineering team
10) license
-
This lists some of the key items recorded inside the license file and current usage
11) memory
-
This lists some data useful in summing up garbage collection events since the server was last started
12) memoryPools
-
this data maps to the same that is shown during a memoryGuard output in the typical log output (every 15 minutes by default)
13) messageServiceStats
-
Some statistics more relevant to the CloudBees Engineering team
14) propertyTracker
-
This section identifies any properties that are currently being monitored
-
These are most typically associated with logic used in a precondition, or a retry setup
15) quartzScheduler
-
Lists schedules that are being monitored for use with Sentry
16) queue
-
Various queues used in the system and their default sizes
17) serverInfo
-
A snapshot of some system settings
18) services
-
Lists the status of the numerous services that sit "under the covers" for the CD system
19) sessionManagerStatistics
-
Data more relevant to the CloudBees Engeering team
20) settings
-
Lists various system setting values
-
This same list can be found on the Server Administration Settings of the UI
21) statistics
-
This section can be beneficial when exploring general performance concerns
-
This section contains 2 sub-sections:
a) All timers by name:
Sample:
Name Count Mean StdDev Min Max Sum ---------------------------------------- ---------- ---------- ---------- ---------- ---------- ------------ setProperty.queue 6692 86.454 399.556 0.000 4292.000 578551.000 setProperty.perform 7715 79.359 124.705 3.000 4869.000 612251.000 setProperty.post 7715 0.147 9.171 0.000 630.000 1132.000 setProperty.last 7715 0.000 0.000 0.000 0.000 0.000
The count represents the number of times this operation occured. The other columns are the times involved, represented in milliseconds.
The columns of most interest tend to be a combination of the Count and Mean entries.
Typically values over 1000 (1s) tend to be a starting point for potential further exploration. However, if the Count for such items is very small, these may be startup operations which are not going to be something of concern, or so rarely executed that they would not be considered a major drag on a running system.
eg: Commands like setProperty taking >1s on average would point to some issue with how the DB is used, or perhaps that the underlying network environment is performing poorly.
From the sample above, typically the .PERFORM number is the one that is most telling of system performance.
If the mean times of numerous operations seem large this may sugget an overall DB issue or system load issue which may require further analysis to uncover ways to mitigate the problem.
b) Timers by roots:
-
This is similar to the section above, but it provides a more detailed breakdown of sub-elements involved with the operation.
-
These extra level of details can be useful in identifying whether the CD software or some 3rd party element may be running imperfectly, or hampered in some way:
eg: Below we can see that the getGroup operation is taking on average over 15s to complete. Howevr, by looking into the breakdown, we also can identify that the bulk of that average time is being spent waiting for the company LDAP service to return with GroupMembers data:
| | | getGroup.perform 92 14884.630 61249.004 0.000 349717.000 1369386.000 | | | | SessionAspect 92 0.000 0.000 0.000 0.000 0.000 | | | | PermissionAspect.validatePermission 84 36.536 212.185 3.000 1489.000 3069.000 | | | | | PermissionAspect.findPermissionAnnotations 84 0.000 0.000 0.000 0.000 0.000 | | | | | PermissionAspect.evaluate 83 0.000 0.000 0.000 0.000 0.000 | | | | | provider.companyLDAP.findFilteredGroups 2 1374.500 146.371 1271.000 1478.000 2749.000 | | | | LicenseAspect 84 0.000 0.000 0.000 0.000 0.000 | | | | provider.companyLDAP.findFilteredGroups 18 1454.889 98.847 1195.000 1573.000 26188.000 | | | | PermissionAspect.evaluate 83 0.000 0.000 0.000 0.000 0.000 | | | | provider.companyLDAP.findGroupMembers 83 16098.325 64135.254 3426.000 348223.000 1336161.000
22) stringExpanderStatistics
-
Typically useful for the CloudBees Engineering team
-
Identifies the number of times a string is being expanded leveraging a cache
23) systemProperties
-
These items map to data picked up at system boot time from configuration files or configured at the time of installation
24) threadDump
-
In cases where a hanging event is seen inside the system, this data may be useful to the CloudBees engineering team
25) transactionRetries
-
Data on the number of transactionRetries in the database
26) transactionState
-
Data on the number of transactions performed by the database
27) versionHistory
-
The list of historical versions that were used on this system prior to the current version in use
-
This can help expose the upgrade path taken with this particular CD server machine
28) messages
-
Typically useful for the CloudBees Engineering team
29) serverState
-
Typically just "running" when viewing this full output
30) serverVersion
-
Details on the current version in use