Summary
Having many agents, many builds, many jobs per builds, and many operations per job can result in vast amounts of data, which complicates agent-side debugging. This article discusses debugging strategies when dealing with large amounts of data.
Solution
ElectricAccelerator provides a number of mechanisms to help you with debugging.
Log rotation
Beginning with version 6.0, Accelerator provides log rotation. Log rotation means that agent-side logs will not grow until they fill up the disk, but instead roll over at a configurable point (see the ElectricAccelerator Installation and Configuration Guide for more details on this feature).
Log rotation configuration options are accessed/configured through the usual agent-side session object (using the logger sub command). Using cmtool is an easy way to control these settings:
cmtool --cm=localhost login admin <*password*>
connects to the locally running Cluster Manager. If you do this interactively, you do not provide the password on the command line, which means cmtool will prompt you. In a scripted setting, for short-term debugging, the command line above is the quickest way to success.
You can control various aspects of the logging:
-
Change the number of log files to retain per agent to 5 (default is 20)
cmtool --cm=localhost runAgentCmd "session logger retaincount 5"
-
Turn off log rotation, clear out the log files, and start log rotation again, respectively
cmtool --cm=localhost runAgentCmd "session logger retaincount 0" cmtool --cm=localhost runAgentCmd "session logger clear" cmtool --cm=localhost runAgentCmd "session logger retaincount 20"
Enabling session tracing
Limiting the total log file size is a good start, but you also want to focus on the real issue, pinpointing the problem. Enable whatever session trace level is requested/needed for debugging. After enabling the appropriate session trace level, all builds will run with that logging level rotating through the logs. The moment you detect the issue (perhaps by seeing that a build failed, or that there is a certain output in the build logs) you call:
cmtool --cm=localhost login admin < password> cmtool --cm=localhost runAgentCmd "session trace log"
which sets tracing back to the default. You could also use "nothing" to turn off tracing completely. Turning off tracing ensures that the last thing in the logs is the error state that is being investigated.
Another way to disable tracing at a predetermined condition is to use an agent-side breakpoint. Refer to the following piece of sample code:
#!/bin/sh
# NOTE: You can use either the supplied php script to track
# retries (see instructions in the file), or use a shared filesystem
# (shared amongst all agents). In that case make sure you adjust the
# fileName variable below.
#
# \
echo "Please enter the Cluster Manager admin password:"
# \
cmtool --cm=localhost login admin
# \
cmtool --cm=localhost runAgentCmd 'session breakpoint set {
if { $breakpoint(location) == "post" && $breakpoint(exitcode) != 0 } {
if {[string match "*Segmentation fault*" $breakpoint(stderr)] ||
[string match "*Segmentation fault*" $breakpoint(stdout)]} {
exec /opt/ecloud/i686_Linux/bin/cmtool --cm=linbuild-cm login admin changeme
exec /opt/ecloud/i686_Linux/bin/cmtool --cm=linbuild-cm runAgentCmd "session trace nothing"
}
}
return 0
# \
} {}'