Flow Log Collector

4 minute readExtensibilityDeveloper productivity

EC-FlowLogCollector is used to collect CloudBees CD/RO and user defined logs from CloudBees CD/RO servers and agents. The plugin can be used in both CloudBees CD/RO standalone servers as well as clusters.

Plugin version 1.1.4.2020092401 Revised on September 17, 2020

Prerequisites

  • This plugin uses the STOMP server for sending and receiving files. Please ensure that STOMP port (61613 by default) on the CloudBees CD Server on which the plugin is installed and executed, is accessible from the CloudBees CD resources from which logs are collected.

To check STOMP server URI, please run the command:

ectool getServerInfo

The example of the output:

<response requestId="1" nodeId="10.200.1.167">
    <serverInfo>
      <cluster>0</cluster>
      <httpPort>8000</httpPort>
      <httpsPort>8443</httpsPort>
      <jobEventsDestination>/topic/events.job</jobEventsDestination>
      <jobEventsSelector>path = '/jobs/$[jobId]'</jobEventsSelector>
      <putFileDestination>/queue/jobs.$[jobId]</putFileDestination>
      <putFileSelector>command='PUT_FILE' AND channel='$[channel]'</putFileSelector>
      <stompClientUri>stomp+ssl://my.server:61613</stompClientUri>
    </serverInfo>
  </response>

To check connectivity, please run the following command from the resource:

telnet my.server 61613

If the telnet command has finished with success, the STOMP server is accessible.

STOMP uri can be configured under server properties:

  1. Go to the Administration > Server

  2. Click on "Settings"

  3. Find the setting "Stomp Client URI"

  4. Set it to stomp+ssl://<accessible hostname>:<STOMP Port>

STOMP server port is configured in commander.properties, under a property called COMMANDER_STOMP_PORT.

  • When using the plugin to collect logs from all the nodes of a Cluster, make sure that if the default pool does not include the local agent resources from each of the nodes, the local agent resources are included explicitly in the parameter list for the "Log Target Resources".

  • The parameter "Filter CD Server Logs by Job Info" requires Linux with zgrep utility installed.

Limitations

  • The plugin uses command line execution to retrieve logs from each CloudBees CD resource. In a Cluster topology where Nodes only have remote agents, CloudBees CD Server logs cannot be collected. In other words any node from which server logs needs to be collected should have a functioning local agent resource.

  • The ability to collect CloudBees CD Server Logs based on filtering by Job Info is not implemented for Windows.

Plugin Procedures

Collect Logs

This procedure collects CloudBees CD logs as well as user defined logs (for example Apache logs, Third Party Tool logs) from both CloudBees CD servers and agents.

Collect Logs Parameters

ParameterDescription

Log Collector Resource

Enter the name of the CloudBees CD Resource that performs the work of log collection. Zip archive containing the collected logs will be created in this resource.

Log Target Resources

Enter Name of comma separated list of CloudBees CD Resources or Resource Pools from which logs need to be collected.

Collect CloudBees CD Server Logs

Collect the commander.log and commander-service.log from the applicable Log Target Resource.

Collect CloudBees CD Agent Logs

Collect the agent.log and jagent.log from the applicable Log Target Resource.

Collect CloudBees CD Repository Logs

Collect the repository.log from the applicable Log Target Resource.

Collect Job Logs

Collect the Job Logs matching the Job IDs provided in the Job ID text area, from the applicable Log Target Resource.

CloudBees CD Job IDs

Provide newline separated job IDs to collect job logs.

Collect Installer Logs

Collect the corresponding CloudBees CD server/agent installer logs viz., installer.log, uninstaller.log, upgrade.log, setupScripts.log from the applicable Log Target Resource.

Collect Apache Logs

Collect the corresponding Apache server logs: access.log, error.log and stdout.log.

Collect User Defined Logs

Collect the user specified log names provided in the "User Defined Logs" text area, from the applicable Log Target Resource. User defined log names need to provided with absolute paths. Wildcards are allowed. For example /var/opt/mydir/x.log or /var/opt/mydir/*.log

User Defined Logs

User defined log names need to provided with absolute paths. Wildcards are allowed. For example /var/opt/mydir/x.log or /var/opt/mydir/*.log

Start Timestamp

Enter either an absolute time stamp as in YYYY-MM-DD HH:MM:SS or a relative time stamp as the number of days before. Valid examples are 2017-09-03, 2017-09-03 22:15:00 and 3. 3 would be interpreted as 3 days before current time.

Default will be 1 day before.

End Timestamp

Enter either an absolute time stamp as in YYYY-MM-DD HH:MM:SS or a relative time stamp as the number of days before. Valid examples are 2017-09-03, 2017-09-03 22:15:00 and 3. 3 would be interpreted as 3 days before current time.

Default will be system time at the time of invocation.

Filter CloudBees CD Server Logs by Job Info

Enter job id or job name to fetch only CloudBees CD server logs that contains them.

Obfuscate Sensitive Data

Choose this option to obfuscate known type of sensitive data such as email, URL and IP.

User Defined Sensitive Data

Provide a REGEX pattern to identify sensitive data other than what is obfuscated out of the box.

For example, Username: \w+ or [\w_=].+SERVICENAME=myhost.example.com\). Several regular expressions are separated by a newline.

Collected Logs Identifier

Procedure will return all collected Logs in a zip file named as YYYY-MM-DD.zip. Add an identifier such as DEV or PROD for further categorization. For example the categorization of DEV would result in a file named as DEV-CloudBees FlowLogs-2018-09-03.zip. Default for the zip file name is CloudBees FlowLogs-YYYY-MM-DD.zip.

After the procedure run, the .zip with logs can be downloaded from Job Summary:

After the procedure has completed, the zip archive containing the collected logs can be downloaded from Job Summary. The zip archive will be located under the workspace in the agent resource where the log collector is run. The path to the zip archive will be <job workspace>/artifacts/<log archive name>.zip.

Release notes

EC-FlowLogCollector 1.1.4

  • The documentation has been migrated to the main documentation site.

EC-FlowLogCollector 1.1.3

  • Renaming to "CloudBees CD".

EC-FlowLogCollector 1.1.2

  • Renaming to "CloudBees"

EC-FlowLogCollector 1.1.1

  • Images in the documentation have been fixed: previously the images have not been shown in Google Chrome.

EC-FlowLogCollector 1.1.0

  • Plugin promotion time has been improved.

EC-FlowLogCollector 1.0.1

  • Added metadata that is required for 9.0 release.

EC-FlowLogCollector 1.0.0

  • Initial Release. For CloudBees CD Releases starting 8.5, this plugin can be accessed from the Service Catalog called "Logs Collection" under the Utility category. For CloudBees CD Releases prior to 8.5 this plugin has to be installed and run directly.