Flow Log Collector plugin

EC-FlowLogCollector is used to collect CloudBees CD/RO and user defined logs from CloudBees CD/RO servers and agents. The plugin can be used in both CloudBees CD/RO standalone servers as well as clusters.

Plugin version 2.0.0.2023101908

Revised on October 19, 2023

Updated Perl version required

This plugin is using an updated version of Perl, cb-perl shell (instead of old ec-perl). This means that it needs CloudBees CD agent version 10.3+ in order to work.

Prerequisites

This plugin uses the STOMP server for sending and receiving files. Please ensure that STOMP port (61613 by default) on the CloudBees CD Server on which the plugin is installed and executed, is accessible from the CloudBees CD resources from which logs are collected.

To check STOMP server URI, run the command:
```
  ectool getServerInfo
```
The example of the output:
```
    <response requestId="1" nodeId="10.200.1.167">
        <serverInfo>
          <cluster>0</cluster>
          <httpPort>8000</httpPort>
          <httpsPort>8443</httpsPort>
          <jobEventsDestination>/topic/events.job</jobEventsDestination>
          <jobEventsSelector>path = '/jobs/$[jobId]'</jobEventsSelector>
          <putFileDestination>/queue/jobs.$[jobId]</putFileDestination>
          <putFileSelector>command='PUT_FILE' AND channel='$[channel]'</putFileSelector>
          <stompClientUri>stomp+ssl://my.server:61613</stompClientUri>
        </serverInfo>
      </response>
```
To check connectivity, please run the following command from the resource:
```
  telnet my.server 61613
```
If the telnet command has finished with success, the STOMP server is accessible.

STOMP uri can be configured under server properties:
1. Go to the Administration > Server
2. Click on "Settings"
3. Find the setting "Stomp Client URI"
4. Set it to stomp+ssl://<accessible hostname>:<STOMP Port>
STOMP server port is configured in commander.properties, under a property called COMMANDER_STOMP_PORT.
When using the plugin to collect logs from all the nodes of a Cluster, make sure that if the default pool does not include the local agent resources from each of the nodes, the local agent resources are included explicitly in the parameter list for the "Log Target Resources".
The parameter "Filter CD Server Logs by Job Info" requires Linux with zgrep utility installed.

Limitations

The plugin uses command line execution to retrieve logs from each CloudBees CD resource. In a Cluster topology where Nodes only have remote agents, CloudBees CD Server logs cannot be collected. In other words any node from which server logs needs to be collected should have a functioning local agent resource.
The ability to collect CloudBees CD Server Logs based on filtering by Job Info is not implemented for Windows.

Plugin procedures

Collect Logs

Collect Logs parameters

Parameter	Description
Log Collector Resource	Required. Enter the name of the CloudBees CD Resource that performs the work of log collection. Zip archive containing the collected logs will be created in this resource.
Log Target Resources	Required. Enter Name of comma separated list of CloudBees CD Resources or Resource Pools from which logs need to be collected.
Collect CloudBees CD Server Logs	Collect the commander.log and commander-service.log from the applicable Log Target Resource.
Collect CloudBees CD Agent Logs	Collect the agent.log and jagent.log from the applicable Log Target Resource.
Collect CloudBees CD Repository Logs	Collect the repository.log from the applicable Log Target Resource.
Collect Job Logs	Collect the Job Logs matching the Job IDs provided in the Job ID text area, from the applicable Log Target Resource.
CloudBees CD Job IDs	Provide newline separated job IDs to collect job logs.
Collect Installer Logs	Collect the corresponding CloudBees CD server/agent installer logs viz., installer.log, uninstaller.log, upgrade.log, setupScripts.log from the applicable Log Target Resource.
Collect Apache Logs	Collect the corresponding Apache server logs: access.log, error.log and stdout.log.
Collect User Defined Logs	Collect the user specified log names provided in the "User Defined Logs" text area, from the applicable Log Target Resource. User defined log names need to provided with absolute paths. Wildcards are allowed. For example /var/opt/mydir/x.log or /var/opt/mydir/*.log
User Defined Logs	User defined log names need to provided with absolute paths. Wildcards are allowed. For example /var/opt/mydir/x.log or /var/opt/mydir/*.log
Start Timestamp	Enter either an absolute time stamp as in YYYY-MM-DD HH:MM:SS or a relative time stamp as the number of days before. Valid examples are 2017-09-03, 2017-09-03 22:15:00 and 3. 3 would be interpreted as 3 days before current time. Default will be 1 day before.
End Timestamp	Enter either an absolute time stamp as in YYYY-MM-DD HH:MM:SS or a relative time stamp as the number of days before. Valid examples are 2017-09-03, 2017-09-03 22:15:00 and 3. 3 would be interpreted as 3 days before current time. Default will be system time at the time of invocation.
Filter CloudBees CD Server Logs by Job Info	Enter job id or job name to fetch only CloudBees CD server logs that contains them.
Obfuscate Sensitive Data	Choose this option to obfuscate known type of sensitive data such as email, URL and IP.
User Defined Sensitive Data	Provide a REGEX pattern to identify sensitive data other than what is obfuscated out of the box. For example, Username: \w+ or [\w_=].+SERVICENAME=myhost.example.com\). Several regular expressions are separated by a newline.
Collected Logs Identifier	Procedure will return all collected Logs in a zip file named as YYYY-MM-DD.zip. Add an identifier such as DEV or PROD for further categorization. For example the categorization of DEV would result in a file named as DEV-CloudBees FlowLogs-2018-09-03.zip. Default for the zip file name is CloudBees FlowLogs-YYYY-MM-DD.zip.
Result property	Provide a property path to store run results. Results are generated to the path provided in the archivePath property.

Parameter

Description

Log Collector Resource

Required. Enter the name of the CloudBees CD Resource that performs the work of log collection. Zip archive containing the collected logs will be created in this resource.

Log Target Resources

Required. Enter Name of comma separated list of CloudBees CD Resources or Resource Pools from which logs need to be collected.

Collect CloudBees CD Server Logs

Collect the commander.log and commander-service.log from the applicable Log Target Resource.

Collect CloudBees CD Agent Logs

Collect the agent.log and jagent.log from the applicable Log Target Resource.

Collect CloudBees CD Repository Logs

Collect the repository.log from the applicable Log Target Resource.

Collect Job Logs

Collect the Job Logs matching the Job IDs provided in the Job ID text area, from the applicable Log Target Resource.

CloudBees CD Job IDs

Provide newline separated job IDs to collect job logs.

Collect Installer Logs

Collect the corresponding CloudBees CD server/agent installer logs viz., installer.log, uninstaller.log, upgrade.log, setupScripts.log from the applicable Log Target Resource.

Collect Apache Logs

Collect the corresponding Apache server logs: access.log, error.log and stdout.log.

Collect User Defined Logs

Collect the user specified log names provided in the "User Defined Logs" text area, from the applicable Log Target Resource. User defined log names need to provided with absolute paths. Wildcards are allowed. For example /var/opt/mydir/x.log or /var/opt/mydir/*.log

User Defined Logs

User defined log names need to provided with absolute paths. Wildcards are allowed. For example /var/opt/mydir/x.log or /var/opt/mydir/*.log

Start Timestamp

Enter either an absolute time stamp as in YYYY-MM-DD HH:MM:SS or a relative time stamp as the number of days before. Valid examples are 2017-09-03, 2017-09-03 22:15:00 and 3. 3 would be interpreted as 3 days before current time.

Default will be 1 day before.

End Timestamp

Default will be system time at the time of invocation.

Filter CloudBees CD Server Logs by Job Info

Enter job id or job name to fetch only CloudBees CD server logs that contains them.

Obfuscate Sensitive Data

Choose this option to obfuscate known type of sensitive data such as email, URL and IP.

User Defined Sensitive Data

Provide a REGEX pattern to identify sensitive data other than what is obfuscated out of the box.

For example, Username: \w+ or [\w_=].+SERVICENAME=myhost.example.com\). Several regular expressions are separated by a newline.

Collected Logs Identifier

Procedure will return all collected Logs in a zip file named as YYYY-MM-DD.zip. Add an identifier such as DEV or PROD for further categorization. For example the categorization of DEV would result in a file named as DEV-CloudBees FlowLogs-2018-09-03.zip. Default for the zip file name is CloudBees FlowLogs-YYYY-MM-DD.zip.

Result property

Provide a property path to store run results. Results are generated to the path provided in the archivePath property.

After the procedure has completed, the zip archive containing the collected logs can be downloaded from Job Summary. The zip archive will be located under the workspace in the agent resource where the log collector is run. The path to the zip archive will be <job workspace>/artifacts/<log archive name>.zip.

Release notes

EC-FlowLogCollector 2.0.0

Upgraded from Perl 5.8 to Perl 5.32. The EC-FlowLogCollector plugin is not backward compatible with CloudBees CD/RO 10.3 and earlier.
Starting with the 2.0.0 release, a new agent is required to run EC-FlowLogCollector plugin procedures.

EC-FlowLogCollector 1.2.0

Added the Result property field to the Gather Logs procedure.

EC-FlowLogCollector 1.1.4

The documentation has been migrated to the main documentation site.

EC-FlowLogCollector 1.1.3

Renaming from "CloudBees Flow" to "CloudBees CD".

EC-FlowLogCollector 1.1.2

Renaming from "Electric Cloud" to "CloudBees"

EC-FlowLogCollector 1.1.1

Images in the documentation have been fixed: previously the images have not been shown in Google Chrome.

EC-FlowLogCollector 1.1.0

Plugin promotion time has been improved.

EC-FlowLogCollector 1.0.1

Added metadata that is required for 9.0 release.

EC-FlowLogCollector 1.0.0

Initial Release. For CloudBees CD Releases starting 8.5, this plugin can be accessed from the Service Catalog called "Logs Collection" under the Utility category. For CloudBees CD Releases prior to 8.5 this plugin has to be installed and run directly.