Missing data on Operations Center Analytics

Article ID:360002690452
2 minute readKnowledge base

Issue

  • We detected that some build data is not reported to Operations Center Analytics

  • We saw the following message in CJP-OC/CJE-OC logs

Mar 27, 2018 11:39:44 AM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue readChanges
The item does not exist anymore Update{type=update, item=null, locator= ... }

  • We saw the following message in CJP-CM/CJE-MM logs

Apr 06, 2018 12:20:53 PM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue submitQueueItems
Failed to submit changes
java.io.IOException: Unexpected return code calling items : 500
    at com.cloudbees.opscenter.analytics.reporter.async.AsyncReporter$1.onCompleted(AsyncReporter.java:78)
    at com.cloudbees.opscenter.analytics.reporter.async.AsyncReporter$1.onCompleted(AsyncReporter.java:69)
    at com.ning.http.client.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:63)
    at com.ning.http.client.providers.netty.NettyResponseFuture.getContent(NettyResponseFuture.java:279)
    at com.ning.http.client.providers.netty.NettyResponseFuture.done(NettyResponseFuture.java:307)

  • We see this Administrative monitor active in CJP-OC/CJE-OC

    es red monitor

  • We see this Administrative monitor active in CJP-OC/CJE-OC/CJP-CM/CJE-MM

    error 500 metrics cm

Environment

Resolution

Potential causes and how to find each one:

Elasticsearch is offline

If you see the Administrative monitor like the two below in CJP-OC/CJE-OC, then your Elasticsearch is in red status or offline. Please follow our troubleshooting guide - Elasticsearch Cluster Health

es red monitor
error 500 metrics

Client controllers unable to reach CJOC

If you see the following Administrative monitor active in CJP-CM/CJE-MM, you have to check your CJP-OC/CJE-OC for the Administrative monitor described before. If none are active in Operations Center, then you have to check the connectivity between CJP-CM/CJE-MM and CJP-OC/CJE-OC, see Operations Center-Client controller connectivity issues

error 500 metrics cm

Missing build data

If you are missing some build data or you saw the following message in your CJP-CM/CJE-MM logs:

Mar 27, 2018 11:39:44 AM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue readChanges
The item does not exist anymore Update{type=update, item=null, locator= ... }

You should check the build history of your jobs. Only the existing items are reported to Operations Center Analytics which means that when the analytics reporter queue sends the data to CJP-OC/CJE-OC the related item should exist. Because of that, it is recommended to keep builds for at least 24 hours or the maximum period that you would have an outage in your infrastructure. Thus if you keep builds for at least 24 hours and Elasticsearch or CJOC is offline for this period, you would not lose builds which are rotated during the outage.

Deleting Old Builds - Best Strategy for Cleanup and disk space management

Missing Nodes data

If you are missing some nodes data or you saw the following message in your CJP-CM/CJE-MM logs:

Mar 27, 2018 11:39:44 AM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue readChanges
The item does not exist anymore Update{type=update, item=null, locator= ... }

Some types of Agents are not permanent items in your Client/Managed controller, e.g., Share Agents, Shared Cloud, or One-Shot-executors. These Agents are used for a short period and then removed. If the time between transient Agent creation and deletion is shorter than one minute, then analytics reporter queue would not find the item and would not be able to report the data to Operations Center Analytics.