You can use breakpoints to help debug hard-to-isolate build issues. Two types of breakpoints exist:
-
The ecbreakpoint application used within makefiles
-
Agent-side breakpoints written in Tcl that are triggered by the agent
The ecbreakpoint
program is a client-server application written in C. It can be called from within makefile rule bodies, shell scripts, or from agent-side breakpoints. When ecbreakpoint
runs, it sends an “I’m waiting” message to the Cluster Manager over an HTTP channel and creates a socket and sits in a loop waiting for shell or agent commands. In addition, you can interact with ecbreakpoint
using curl
or wget
. It is important to note that only the job is stopped, not the entire build. Other jobs continue to be scheduled and run, including jobs on other agents on the same cluster host.
The agent-side breakpoint feature is more difficult to configure and debug, but it enables a higher level of precision when trying to find problems. The agent-side breakpoint enables agents to run arbitrary commands when specific conditions are encountered. Typically, the Tcl breakpoint code calls ecbreakpoint
when the stdout from a job matches a particular string. However, any command can be run, including shell scripts and tools such as lsof
and strace/truss
on UNIX and procmon
on Windows.
After ecbreakpoint
stops a job, you can use the Cluster Manager Break Point Details page to send arbitrary shell and session commands to the agent. In addition, you can rerun or continue the job. You must have full breakpoint permissions to use all breakpoint operations. If you cannot see breakpoints, you must change your permissions. Go to Administration > Permissions > Edit Permissions (for your account) > Break Points: Full
By stopping jobs, you can query the agents or operating system in the context that a job ran. Scenarios where you might want to use the breakpoint feature:
-
Files are not available when they should be
-
Permission problems
-
Perform checksums on files
-
Examine an agent’s state
Using ecbreakpoint applications
The ecbreakpoint
executable ( ecbreakpoint
/ ecbreakpoint.exe
) resides in the agent machine’s ecloud bin directory (for example, ` /opt/ecloud/i686/bin/` on Linux, and c:\ecloud\i686_win32\bin\
on Windows). Table 1 shows `ecbreakpoint ` command line arguments.
Usage: |
|
` --cm=<value> ` |
Name of the Cluster Manager to connect to |
`-d, --debug ` |
Enable debug mode |
|
Do not connect to the Cluster Manager |
|
This usage message |
|
Label for the breakpoint. This is used in the web UI to identify the breakpoint |
|
HTTP port to use to contact this application |
|
Print the label when the breakpoint is reached |
|
Number of seconds after which the breakpoint continues. The default is 10 days |
|
Print version information |
Table 1: ecbreakpoint
command-line options
Using ecbreakpoint
You can add ecbreakpoint
to makefile rule bodies as in Figure 1. eMake pauses the job when it encounters the breakpoint. The Cluster Manager then displays the breakpoint panel on the Build Details page (Figure 3). Click a breakpoint action to display the Break Point Detail page.
|
Figure 1: Makefile using the ecbreakpoint
program
Figure 2 illustrates a build (chronic_641) with jobs paused by ecbreakpoint
.
Figure 2: Paused job
Figure 3: Breakpoint UI on the Build Details page
If you have sufficient permissions , you can interact with the ecbreakpoint
program within the web interface. There are currently four actions available for ecbreakpoint
:
-
Retry —Redo this job step from the beginning
-
Continue —Step past the breakpoint
-
Agent Command —Issue agent commands, such as
session state
, to the stopped agent -
Shell Command —Issue shell commands to the stopped agent. These commands run in the same context (filesystem, environment) as the job itself. For UNIX, the bash shell is used to execute commands; for Windows, cmd.exe is used. Therefore,
;
is not allowed in theecbreakpoint
shell command for Windows.
Click Retry or Continue to delete the current ecbreakpoint
.
Click Agent Command or Shell Command to go to the Break Point Details page (illustrated in Figure 4), which provides more information and interaction with the breakpoint. After you issue a command, the response is shown as Waiting for response…
. Refresh the page after a few seconds to view the result (the lower portion of Figure 4). The value of the request Type column can be S or A, which is shell or agent command respectively.
Figure 4 illustrates that pstree -alp
was passed to the agent. The pstree
command shows a hierarchical representation of the process tree and is often useful to see what commands agents are running. Note that pstree
is a child of ecbreakpoint
and ecbreakpoint
is a child of agent 0. The compilation and linking processes will probably be finished by the time you run pstree
. You may see other processes, for example, processes launched in the background such as license managers, that may provide clues about what is happening.
Figure 4: ` pstree` command run from the breakpoint menu
You can also get session state information to send to Electric Cloud technical support. For contact information, see https://support.cloudbees.com/hc/en-us/categories/360002059512 .
Figure 5: Session state information
You can access ecbreakpoint
from the command line or scripts using the curl command. CloudBees Build Acceleration ships with curl in the ecloud bin directory. Figure 3 shows the IP address and port for ecbreakpoint
. Use those values in curl and make sure to use the appropriate escape characters for spaces, slashes, and so on.
Following is an example for running the ps
command on the agent:
https://192.168.26.235:41184/exec.xml?command=%2Fbi2Fps%20-ef
Table 2 contains URLs you can use to access ecbreakpoint
.
|
Rerun the command on a different agent. |
|
Continue normal execution. |
|
Execute the specified command. The command is one simple command passed with the command GET parameter. |
|
Execute the specified agent command. The command is one simple command passed with the command GET parameter. |
|
Show the current agent session state. |
Table 2: ecbreakpoint
access URLs
Labeling breakpoints
The makefile in Figure 1 contained one ecbreakpoint
command. If you want to set multiple breakpoints in your build, you must use the ecbreakpoint --label
command line option to distinguish them in the UI. Notice the --label
option in the Makefile in Figure 6.
|
Figure 6: Makefile with a label for each target
When the Makefile in Figure 6 runs, the Cluster Manager displays labels for each breakpoint (Figure 7).
Figure 7: Labeled breakpoints on the Build Details page
The label command may need more information than just the target name. For example, if a pattern rule was called multiple times from various submakes where the C files had the same names, this could result in a situation similar to the one in Figure 8 where the labels are the same.
Figure 8: Breakpoint labels with the same name
You can make labels more distinct by specifying the label as --label="$(@) \_pwd"
, which results in the output in Figure 10.
|
Figure 9: ` ecbreakpoint` label option augmented with the working directory
Figure 10: Breakpoint labels with the working directory
Using agent-side breakpoints
You can use agent-side breakpoints when:
-
you do not know where in the makefile a problem occurs, but you want to stop the build based on some condition
-
you know, after the fact, where a problem occurs in a makefile, but the location keeps changing
You can use agent-side breakpoints to configure agents to run arbitrary commands when a particular condition is found. The arbitrary command can be ecbreakpoint
, but it does not have to be. You can use the cmtool runAgentCmd
program to configure agent-side breakpoints (Figure 1).
|
Figure 1: Session commands to manipulate agent-side breakpoints
Agent-side breakpoints enable agents to run a block of Tcl code before and after running the job command. The breakpoint consists of a block of trigger code and a command to run if the trigger code returns a non-zero return code. If the trigger block results in a non-zero exit code, the agent runs the command in the second block. The trigger code has access to the read-only breakpoint associative array that contains the entries in Table 1. Generally, the Tcl code you write will use regular expressions to match the stdout element of the breakpoint array. By default, eMake runs with the “merge streams” option so there is usually no reason to look in stderr.
Array | Contents |
---|---|
|
The command line |
|
“pre” or “post”, for before or after the command runs |
|
Exit code of the command; -1 for “pre” checks |
|
Output of the command; empty for “pre” checks |
|
stderr of the command. Generally, mergestreams is enabled, so you can look at stdout. |
Table 1: Breakpoint associative array
Figure 2 illustrates a breakpoint that triggers if the standard output of a command contains the text ` parse error` or ` undeclared identifier`.
|
Figure 2: Agent breakpoint triggered by regex on stdout
When you set the breakpoint and run a makefile that emits the desired error strings (Figure 3), you see the breakpoints displayed on the Build Details page (Figure 4).
|
Figure 3: Makefile to generate error
Using the agent name displayed in the Break Points panel (Figure 4), you can find the output from the agent breakpoint in the agent logs (/var/log/ecagent#.logs ) (Figure 5).
Figure 4: Cluster Manager UI for agent-side breakpoints
|
Figure 5: /var/log/ecagent4.log
Figure 4 illustrates the empty Label column. You cannot use the ecbreakpoint
--label
argument in the Figure 2 breakpoint because the breakpoint is set before jobs are run. Remember that the agent-side breakpoint can run arbitrary commands. The breakpoint code in Figure 6 accesses the command-line string from the breakpoint array, constructs a shell script that calls ecbreakpoint
with the --label
you create, and then runs the shell script when the breakpoint triggers.
|
Figure 6: Agent breakpoint that initiates an arbitrary program
When you run the makefile in Figure 3 with the new breakpoint code, more descriptive labels are printed in the Break Points panel (Figure 7).
Figure 7: Breakpoint list on the Build Details page
Because the job path is not available to the agents, the example uses the command name as an identifier even though it is a poor substitute for the job path and might not always be unique. The jobid is available, but it does not correlate to the annotation file at the point in time when the breakpoint triggers because job details are not written to the annotation file until termination time. |
Specifying a breakpoint through a file
You can also specify the breakpoint by passing in a file. Begin the command argument with file: <yourpath>
and the file will be read from the path specified for the runAgentCmd request.
Example breakpoint contained in a file named test.txt
:
|
Example of passing in the breakpoint contained in test.txt
:
|
Troubleshooting breakpoints
After you create an agent-side breakpoint, it might not work as expected or provide any output. following are possible problems and solutions:
Breakpoint does not trigger
If the breakpoint does not trigger when you think it should, look in the agent log files. If there are Tcl errors with your breakpoint, you may see something like the following:
[228/0] 0.081472/105997992.238145 SYSTEM_LOG: INFO Error evaluating breakpoint condition: can't read "agentId": no such variableTcl errors will prevent the breakpoint from running.
Breakpoint regex is wrong
Be sure that the text you are looking for actually appears in the stdout. If the text is in the output, open the tkcon console (in the ecloud bin directory) and test your regex code. Modify the regex until it works.
Output is not as expected
Output from agent-side breakpoint 'puts' statements is written to agent logs and not the build’s stdout. If the agent breakpoint stops the build with ecbreakpoint
, you can use the Break Points panel to determine which agent log to examine. However, if your breakpoint calls a tool such as strace
, you may have to examine all agent logs. On Linux, one useful hint to remember is that Gnu ` tail(1)` can tail multiple files at the same time ( tail -f /var/log/ecagent?.log
).
Use your editor to run breakpoints
When creating agent breakpoints, it is useful to have an open editor window with the cmtool runAgentCommand set breakpoint
command that you are modifying. When working with the breakpoint, you can execute it from within the editor by selecting the code in Vim’s visual mode and then using the 'ex' command line to feed it into the shell for execution. You do not need to clear the breakpoint before setting a new one because setting a breakpoint overwrites the current one. You can also cut-and-paste the breakpoint code into a command window.
In the .vimrc
, configure the makeprg
variable to call eMake instead of GNU Make; type ` :make` from the ex command line to invoke the makefile.
set makeprg=/opt/ecloud/i686_Linux/bin/emake\ --emake-cm=support-lin1
In emacs, you can feed text to the shell using 'M-|'. Select the region of the buffer you want to send to the shell, and then type M-|.