Troubleshooting CloudBees CD/RO jobs stuck in Running state due to ActiveMQ disk usage threshold

Last Reviewed:2026-07-13()
2 minute readKnowledge base

Symptoms

Users may experience one or more of the following:

  • Running jobs become unresponsive and remain stuck in the Running state indefinitely

  • The Abort command does not respond or has no effect on stuck jobs

  • commander.log contains repeated Deadlock found when trying to get lock and Transaction timed out errors

  • commander.log contains ActiveMQ warnings such as:

    AMQ222210: Free storage space is at 84.8GB of 959.0GB total. Usage rate is 91.2% which is beyond the configured <max-disk-usage>. System will start blocking producers.
    ▼
    AMQ222212: Disk Full! Blocking message production on address: /topic/events.job. Clients will not be able to produce messages until disk space is freed.
    ▼
    AMQ212054: Destination is blocked: /topic/events.job
    ▼

  • Internal event topics such as /topic/events.job and /topic/events.flowRuntime are blocked

Diagnosis/Treatment

Check disk usage on the CloudBees CD/RO server

  1. Check current disk usage on the host running CloudBees CD/RO:

    df -h
    ▼

  2. If disk usage is at or above 90%, the embedded ActiveMQ Artemis broker will have triggered Producer Flow Control, blocking internal event message production.

  3. This blocks job state transitions and causes the symptoms listed above.

Confirm ActiveMQ Producer Flow Control is triggered

  1. Open commander.log and search for the following error codes:

    AMQ222210
AMQ222212
AMQ212054

  2. The presence of AMQ222210 or AMQ222212 confirms that the broker detected disk usage above the configured <max-disk-usage> threshold (default: 90%).

  3. The presence of AMQ212054 confirms that message production on internal topics (for example, /topic/events.job) is actively blocked.

The /topic/events.job and /topic/events.flowRuntime destinations are virtual internal ActiveMQ topics used by CloudBees CD/RO for job state updates and flow runtime events. They are not filesystem paths.

Resolve by freeing disk space and restarting the server

If the above diagnostics confirm disk usage is above the 90% threshold:

  1. Identify and remove unnecessary files from the CloudBees CD/RO server host to reduce disk usage below 90%. Common candidates include old log files, temporary files, and archived data.

  2. Verify that disk usage has dropped below the 90% threshold:

    df -h
    ▼

  3. Restart the CloudBees CD/RO server to clear the blocked ActiveMQ producer state.

  4. After restart, confirm that previously stuck jobs are now progressing or can be aborted successfully.

Simply freeing disk space without restarting the CloudBees CD/RO server may not immediately unblock the ActiveMQ broker. A server restart is required to restore normal operation.

Understand the max-disk-usage configuration

The embedded ActiveMQ Artemis broker uses a percentage-based disk threshold configured in broker.xml:

<max-disk-usage>90</max-disk-usage>
▼

When disk usage reaches or exceeds this percentage, the broker stops accepting new messages from internal producers, which prevents job state updates from being processed.

Lowering the <max-disk-usage> value raises the sensitivity of flow control but does not eliminate the risk of disk exhaustion. Changing this value is not recommended unless you have deep ActiveMQ Artemis expertise, as it acts as a stability safeguard for the broker. The recommended approach is to implement proactive disk space monitoring to prevent the threshold from being reached.

Prevent recurrence with disk space monitoring

To prevent this issue from recurring:

  1. Implement OS-level or infrastructure monitoring to alert when disk usage on the CloudBees CD/RO server host exceeds 75–80%, providing time to act before the 90% broker threshold is reached.

  2. Review and enforce log retention and data archival policies to control disk consumption over time.

  3. Consider increasing storage capacity if disk usage regularly approaches the threshold.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.