Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction
Introduction

Flow Log Collector plugin

4 minute readExtensibilityDeveloper productivity

The Flow Log Collector plugin is used to collect CloudBees CD/RO and user-defined logs from CloudBees CD/RO servers and agents. The plugin can be used in both CloudBees CD/RO standalone servers, as well as clusters.

Plugin version 2.0.0.2023101908

Revised on October 19, 2023

Prerequisites

  • This plugin is using an updated version of Perl with the cb-perl shell, and requires CloudBees CD/RO agent version 10.3+ in order to work.

  • This plugin uses the STOMP server for sending and receiving files. Please ensure that the STOMP port (61613, by default) on the CloudBees CD/RO server where the plugin is installed and executed, is accessible from the CloudBees CD/RO resources from which logs are collected.

    To check STOMP server URI, run the following command:

      ectool getServerInfo
    ▼

    The output should look similar to the following:

        <response requestId="1" nodeId="10.200.1.167">
        <serverInfo>
          <cluster>0</cluster>
          <httpPort>8000</httpPort>
          <httpsPort>8443</httpsPort>
          <jobEventsDestination>/topic/events.job</jobEventsDestination>
          <jobEventsSelector>path = '/jobs/$[jobId]'</jobEventsSelector>
          <putFileDestination>/queue/jobs.$[jobId]</putFileDestination>
          <putFileSelector>command='PUT_FILE' AND channel='$[channel]'</putFileSelector>
          <stompClientUri>stomp+ssl://my.server:61613</stompClientUri>
        </serverInfo>
      </response>
    ▼

    To check connectivity, run the following command from the resource:

      telnet my.server 61613
    ▼

    If the telnet command is successful, the STOMP server is accessible.

    The STOMP URI can be configured under server properties:

    1. Navigate to Administration  Server settings  System Settings.

    2. Locate the Stomp Client URI setting.

    3. Set it to stomp+ssl://<accessible hostname>:<STOMP Port>

      The STOMP server port is configured in commander.properties, under a property named COMMANDER_STOMP_PORT.

  • When using the plugin to collect logs from all the nodes of a Cluster, ensure that if the default pool does not include the local agent resources from each of the nodes, the local agent resources are included explicitly in the parameter list for the Log target resources.

  • The Filter CD/RO server logs by job info parameter requires Linux with the zgrep utility installed.

Limitations

  • The plugin uses command line execution to retrieve logs from each CloudBees CD/RO resource. In a cluster topology where nodes only have remote agents, CloudBees CD/RO server logs cannot be collected. In other words, any node from which the server logs needs to be collected should have a functioning local agent resource.

  • The ability to collect CloudBees CD/RO server logs based on filtering by Job Info is not implemented for Windows.

Plugin procedures

Collect Logs

Collect Logs parameters

Parameter Description

Log collector resource

CloudBees CD/RO resource that performs the work of log collection. A .zip archive containing the collected logs is created in this resource.

Log target resources

Comma-separated list of CloudBees CD/RO resources or resource pools from which the logs need to be collected.

Collect CloudBees CD/RO server logs

Collect the commander.log and commander-service.log files from the applicable log target resource.

Collect CloudBees CD/RO agent logs

Collect the agent.log and jagent.log files from the applicable log target resource.

Collect CloudBees CD/RO repository logs

Collect the repository.log from the applicable log target resource.

Collect job logs

Collect the job logs matching the job IDs provided in the CloudBees CD/RO job IDs field, from the applicable log target resource.

CloudBees CD/RO job IDs

Newline-separated job IDs to collect job logs.

Collect installer logs

Collect the corresponding CloudBees CD/RO server/agent installer logs: viz.log, installer.log, uninstaller.log, upgrade.log, and setupScripts.log files from the applicable log target resource.

Collect Apache logs

Collect the corresponding Apache server logs: access.log, error.log, and stdout.log files from the applicable log target resource.

Collect user-defined logs

Collect the user-specified log names specified in the User-defined logs field. User-defined log names need to provided with absolute paths. Wildcards are allowed. For example, /var/opt/mydir/x.log or /var/opt/mydir/*.log.

User-defined logs

User-defined log names need to provided with absolute paths. Wildcards are allowed. For example, /var/opt/mydir/x.log or /var/opt/mydir/*.log.

Start timestamp

Enter either an absolute timestamp in the format: YYYY-MM-DD HH:MM:SS or a relative timestamp as the number of days before. Valid examples are: 2017-09-03, 2017-09-03 22:15:00, and 3. 3 is interpreted as 3 days before the current time.

The default is 1 day before.

End timestamp

Enter either an absolute timestamp in the format: YYYY-MM-DD HH:MM:SS or a relative timestamp as the number of days before. Valid examples are 2017-09-03, 2017-09-03 22:15:00, and 3. 3 is interpreted as 3 days before the current time.

The default is the system time at the time of invocation.

Filter CloudBees CD/RO server logs by job info

Enter a job ID or job name to fetch only CloudBees CD/RO server logs that contain the job ID or name.

Obfuscate sensitive data

Select to obfuscate known types of sensitive data such as email, URL, and IP address.

User-defined sensitive data

Comma-separated list of regular expressions to identify sensitive data other than what is obfuscated by default. For example, Username: \w+ or [\w_=].+SERVICENAME=myhost.example.com\.

Collected logs identifier

Returns all collected logs in a .zip file named in the format: YYYY-MM-DD.zip. Add an identifier such as DEV or PROD for further categorization. For example, the categorization of DEV would result in a file named DEV-CloudBees FlowLogs-2018-09-03.zip. The default .zip file name is CloudBees FlowLogs-YYYY-MM-DD.zip.

Result property

Provide a property path to store run results. Results are generated to the path provided in the archivePath property.

Release notes

EC-FlowLogCollector 2.0.0

  • Upgraded from Perl 5.8 to Perl 5.32. The EC-FlowLogCollector plugin is not backward compatible with CloudBees CD/RO 10.3 and earlier.

  • Starting with the 2.0.0 release, a new agent is required to run EC-FlowLogCollector plugin procedures.

EC-FlowLogCollector 1.2.0

  • Added the Result property field to the Gather Logs procedure.

EC-FlowLogCollector 1.1.4

  • The documentation has been migrated to the main documentation site.

EC-FlowLogCollector 1.1.3

  • Renaming from "CloudBees Flow" to "CloudBees CD".

EC-FlowLogCollector 1.1.2

  • Renaming from "Electric Cloud" to "CloudBees"

EC-FlowLogCollector 1.1.1

  • Images in the documentation have been fixed: previously the images have not been shown in Google Chrome.

EC-FlowLogCollector 1.1.0

  • Plugin promotion time has been improved.

EC-FlowLogCollector 1.0.1

  • Added metadata that is required for the CloudBees CD/RO 9.0 release.

EC-FlowLogCollector 1.0.0

  • Initial Release. For ElectricFlow releases starting with version 8.5, this plugin can be accessed from the Service Catalog named Logs Collection, under the Utility category. For ElectricFlow releases prior to version 8.5, this plugin has to be installed and run directly.