You can use breakpoints to help debug hard-to-isolate build issues. Two types of breakpoints exist:
ecbreakpoint program is a client-server application written in C. It can be called from within makefile rule bodies, shell scripts, or from agent-side breakpoints. When
ecbreakpoint runs, it sends an “I’m waiting” message to the Cluster Manager over an HTTP channel and creates a socket and sits in a loop waiting for shell or agent commands. In addition, you can interact with
wget. It is important to note that only the job is stopped, not the entire build. Other jobs continue to be scheduled and run, including jobs on other agents on the same cluster host.
The agent-side breakpoint feature is more difficult to configure and debug, but it enables a higher level of precision when trying to find problems. The agent-side breakpoint enables agents to run arbitrary commands when specific conditions are encountered. Typically, the Tcl breakpoint code calls
ecbreakpoint when the stdout from a job matches a particular string. However, any command can be run, including shell scripts and tools such as
strace/truss on UNIX and
procmon on Windows.
ecbreakpoint stops a job, you can use the Cluster Manager Break Point Details page to send arbitrary shell and session commands to the agent. In addition, you can rerun or continue the job. You must have full breakpoint permissions to use all breakpoint operations. If you cannot see breakpoints, you must change your permissions. Go to Administration > Permissions > Edit Permissions (for your account) > Break Points: Full
By stopping jobs, you can query the agents or operating system in the context that a job ran. Scenarios where you might want to use the breakpoint feature:
Files are not available when they should be
Perform checksums on files
Examine an agent’s state
ecbreakpoint executable (
ecbreakpoint.exe ) resides in the agent machine’s ecloud bin directory (for example, ` /opt/ecloud/i686/bin/` on Linux, and
c:\ecloud\i686_win32\bin\ on Windows). Table 1 shows `ecbreakpoint ` command line arguments.
` --cm=<value> `
Name of the Cluster Manager to connect to
`-d, --debug `
Enable debug mode
Do not connect to the Cluster Manager
This usage message
Label for the breakpoint. This is used in the web UI to identify the breakpoint
HTTP port to use to contact this application
Print the label when the breakpoint is reached
Number of seconds after which the breakpoint continues. The default is 10 days
Print version information
Table 1: ` ecbreakpoint` command-line options
You can add
ecbreakpoint to makefile rule bodies as in Figure 1. eMake pauses the job when it encounters the breakpoint. The Cluster Manager then displays the breakpoint panel on the Build Details page (Figure 3). Click a breakpoint action to display the Break Point Detail page.
Figure 1: Makefile using the
Figure 2 illustrates a build (chronic_641) with jobs paused by
Figure 2: Paused job
Figure 3: Breakpoint UI on the Build Details page
If you have sufficient permissions , you can interact with the
ecbreakpoint program within the web interface. There are currently four actions available for
Retry —Redo this job step from the beginning
Continue —Step past the breakpoint
Agent Command —Issue agent commands, such as
session state, to the stopped agent
Shell Command —Issue shell commands to the stopped agent. These commands run in the same context (filesystem, environment) as the job itself. For UNIX, the bash shell is used to execute commands; for Windows, cmd.exe is used. Therefore,
;is not allowed in the
ecbreakpointshell command for Windows.
Click Retry or Continue to delete the current
Click Agent Command or Shell Command to go to the Break Point Details page (illustrated in Figure 4), which provides more information and interaction with the breakpoint. After you issue a command, the response is shown as
Waiting for response…. Refresh the page after a few seconds to view the result (the lower portion of Figure 4). The value of the request Type column can be S or A, which is shell or agent command respectively.
Figure 4 illustrates that
pstree -alp was passed to the agent. The
pstree command shows a hierarchical representation of the process tree and is often useful to see what commands agents are running. Note that
pstree is a child of
ecbreakpoint is a child of agent 0. The compilation and linking processes will probably be finished by the time you run
pstree. You may see other processes, for example, processes launched in the background such as license managers, that may provide clues about what is happening.
Figure 4: ` pstree` command run from the breakpoint menu
You can also get session state information to send to Electric Cloud technical support. For contact information, see https://support.cloudbees.com/hc/en-us/categories/360002059512 .
Figure 5: Session state information
You can access
ecbreakpoint from the command line or scripts using the curl command. Accelerator ships with curl in the ecloud bin directory. Figure 3 shows the IP address and port for
ecbreakpoint. Use those values in curl and make sure to use the appropriate escape characters for spaces, slashes, and so on.
Following is an example for running the
ps command on the agent:
Table 2 contains URLs you can use to access
Rerun the command on a different agent.
Continue normal execution.
Execute the specified command. The command is one simple command passed with the command GET parameter.
Execute the specified agent command. The command is one simple command passed with the command GET parameter.
Show the current agent session state.
ecbreakpoint access URLs
The makefile in Figure 1 contained one
ecbreakpoint command. If you want to set multiple breakpoints in your build, you must use the ecbreakpoint
--label command line option to distinguish them in the UI. Notice the
--label option in the Makefile in Figure 6.
Figure 6: Makefile with a label for each target
When the Makefile in Figure 6 runs, the Cluster Manager displays labels for each breakpoint (Figure 7).
Figure 7: Labeled breakpoints on the Build Details page
The label command may need more information than just the target name. For example, if a pattern rule was called multiple times from various submakes where the C files had the same names, this could result in a situation similar to the one in Figure 8 where the labels are the same.
Figure 8: Breakpoint labels with the same name
You can make labels more distinct by specifying the label as ` --label=$(@)_`pwd`` which results in the output in Figure 10.
Figure 9: ` ecbreakpoint` label option augmented with the working directory
Figure 10: Breakpoint labels with the working directory
You can use agent-side breakpoints when:
you do not know where in the makefile a problem occurs, but you want to stop the build based on some condition
you know, after the fact, where a problem occurs in a makefile, but the location keeps changing
You can use agent-side breakpoints to configure agents to run arbitrary commands when a particular condition is found. The arbitrary command can be
ecbreakpoint, but it does not have to be. You can use the
cmtool runAgentCmd program to configure agent-side breakpoints (Figure 1).
Figure 1: Session commands to manipulate agent-side breakpoints
Agent-side breakpoints enable agents to run a block of Tcl code before and after running the job command. The breakpoint consists of a block of trigger code and a command to run if the trigger code returns a non-zero return code. If the trigger block results in a non-zero exit code, the agent runs the command in the second block. The trigger code has access to the read-only breakpoint associative array that contains the entries in Table 1. Generally, the Tcl code you write will use regular expressions to match the stdout element of the breakpoint array. By default, eMake runs with the “merge streams” option so there is usually no reason to look in stderr.
The command line
“pre” or “post”, for before or after the command runs
Exit code of the command; -1 for “pre” checks
Output of the command; empty for “pre” checks
stderr of the command. Generally, mergestreams is enabled, so you can look at stdout.
Table 1: Breakpoint associative array
Figure 2 illustrates a breakpoint that triggers if the standard output of a command contains the text ` parse error` or ` undeclared identifier`.
Figure 2: Agent breakpoint triggered by regex on stdout
When you set the breakpoint and run a makefile that emits the desired error strings (Figure 3), you see the breakpoints displayed on the Build Details page (Figure 4).
Figure 3: Makefile to generate error
Using the agent name displayed in the Break Points panel (Figure 4), you can find the output from the agent breakpoint in the agent logs (/var/log/ecagent#.logs ) (Figure 5).
Figure 4: Cluster Manager UI for agent-side breakpoints
Figure 5: /var/log/ecagent4.log
Figure 4 illustrates the empty Label column. You cannot use the
--label argument in the Figure 2 breakpoint because the breakpoint is set before jobs are run. Remember that the agent-side breakpoint can run arbitrary commands. The breakpoint code in Figure 6 accesses the command-line string from the breakpoint array, constructs a shell script that calls
ecbreakpoint with the
--label you create, and then runs the shell script when the breakpoint triggers.
Figure 6: Agent breakpoint that initiates an arbitrary program
When you run the makefile in Figure 3 with the new breakpoint code, more descriptive labels are printed in the Break Points panel (Figure 7).
Figure 7: Breakpoint list on the Build Details page
|Because the job path is not available to the agents, the example uses the command name as an identifier even though it is a poor substitute for the job path and might not always be unique. The jobid is available, but it does not correlate to the annotation file at the point in time when the breakpoint triggers because job details are not written to the annotation file until termination time.|
You can also specify the breakpoint by passing in a file. Begin the command argument with
file: <yourpath> and the file will be read from the path specified for the runAgentCmd request.
Example breakpoint contained in a file named
Example of passing in the breakpoint contained in
After you create an agent-side breakpoint, it might not work as expected or provide any output. following are possible problems and solutions:
If the breakpoint does not trigger when you think it should, look in the agent log files. If there are Tcl errors with your breakpoint, you may see something like the following:
[228/0] 0.081472/105997992.238145 SYSTEM_LOG: INFO Error evaluating breakpoint condition: can't read "agentId": no such variableTcl errors will prevent the breakpoint from running.
Be sure that the text you are looking for actually appears in the stdout. If the text is in the output, open the tkcon console (in the ecloud bin directory) and test your regex code. Modify the regex until it works.
Output from agent-side breakpoint 'puts' statements is written to agent logs and not the build’s stdout. If the agent breakpoint stops the build with
ecbreakpoint, you can use the Break Points panel to determine which agent log to examine. However, if your breakpoint calls a tool such as
strace, you may have to examine all agent logs. On Linux, one useful hint to remember is that Gnu ` tail(1)` can tail multiple files at the same time (
tail -f /var/log/ecagent?.log ).
When creating agent breakpoints, it is useful to have an open editor window with the
cmtool runAgentCommand set breakpoint command that you are modifying. When working with the breakpoint, you can execute it from within the editor by selecting the code in Vim’s visual mode and then using the 'ex' command line to feed it into the shell for execution. You do not need to clear the breakpoint before setting a new one because setting a breakpoint overwrites the current one. You can also cut-and-paste the breakpoint code into a command window.
.vimrc, configure the
makeprg variable to call eMake instead of GNU Make; type ` :make` from the ex command line to invoke the makefile.
set makeprg=/opt/ecloud/i686_Linux/bin/emake\ --emake-cm=support-lin1
In emacs, you can feed text to the shell using 'M-|'. Select the region of the buffer you want to send to the shell, and then type M-|.