Using breakpoints

10 minute read

You can use breakpoints to help debug hard-to-isolate build issues. Two types of breakpoints exist:

The ecbreakpoint program is a client-server application written in C. It can be called from within makefile rule bodies, shell scripts, or from agent-side breakpoints. When ecbreakpoint runs, it sends an “I’m waiting” message to the Cluster Manager over an HTTP channel and creates a socket and sits in a loop waiting for shell or agent commands. In addition, you can interact with ecbreakpoint using curl or wget. It is important to note that only the job is stopped, not the entire build. Other jobs continue to be scheduled and run, including jobs on other agents on the same cluster host.

The agent-side breakpoint feature is more difficult to configure and debug, but it enables a higher level of precision when trying to find problems. The agent-side breakpoint enables agents to run arbitrary commands when specific conditions are encountered. Typically, the Tcl breakpoint code calls ecbreakpoint when the stdout from a job matches a particular string. However, any command can be run, including shell scripts and tools such as lsof and strace/truss on UNIX and procmon on Windows.

After ecbreakpoint stops a job, you can use the Cluster Manager Break Point Details page to send arbitrary shell and session commands to the agent. In addition, you can rerun or continue the job. You must have full breakpoint permissions to use all breakpoint operations. If you cannot see breakpoints, you must change your permissions. Go to Administration > Permissions > Edit Permissions (for your account) > Break Points: Full

By stopping jobs, you can query the agents or operating system in the context that a job ran. Scenarios where you might want to use the breakpoint feature:

  • Files are not available when they should be

  • Permission problems

  • Perform checksums on files

  • Examine an agent’s state

Using ecbreakpoint applications

The ecbreakpoint executable ( ecbreakpoint / ecbreakpoint.exe ) resides in the agent machine’s ecloud bin directory (for example, ` /opt/ecloud/i686/bin/` on Linux, and c:\ecloud\i686_win32\bin\ on Windows). Table 1 shows `ecbreakpoint ` command line arguments.

Usage: ecbreakpoint [switches]

` --cm=<value> `

Name of the Cluster Manager to connect to

`-d, --debug `

Enable debug mode

--dontConnectToCM

Do not connect to the Cluster Manager

-h, --help

This usage message

--label=<value>

Label for the breakpoint. This is used in the web UI to identify the breakpoint

--port=<value>

HTTP port to use to contact this application

--printLabel

Print the label when the breakpoint is reached

--timeout=<value>

Number of seconds after which the breakpoint continues. The default is 10 days

-v, --version

Print version information

Table 1: ecbreakpoint command-line options

Using ecbreakpoint

You can add ecbreakpoint to makefile rule bodies as in Figure 1. eMake pauses the job when it encounters the breakpoint. The Cluster Manager then displays the breakpoint panel on the Build Details page (Figure 3). Click a breakpoint action to display the Break Point Detail page.

all: a b @ echo Running $@ a: @ echo Before breakpoint ecbreakpoint b: @ echo Some other job

Figure 1: Makefile using the ecbreakpoint program

Figure 2 illustrates a build (chronic_641) with jobs paused by ecbreakpoint.

Figure 2: Paused job

Figure 3: Breakpoint UI on the Build Details page

If you have sufficient permissions , you can interact with the ecbreakpoint program within the web interface. There are currently four actions available for ecbreakpoint :

  • Retry —Redo this job step from the beginning

  • Continue —Step past the breakpoint

  • Agent Command —Issue agent commands, such as session state, to the stopped agent

  • Shell Command —Issue shell commands to the stopped agent. These commands run in the same context (filesystem, environment) as the job itself. For UNIX, the bash shell is used to execute commands; for Windows, cmd.exe is used. Therefore, ; is not allowed in the ecbreakpoint shell command for Windows.

Click Retry or Continue to delete the current ecbreakpoint.

Click Agent Command or Shell Command to go to the Break Point Details page (illustrated in Figure 4), which provides more information and interaction with the breakpoint. After you issue a command, the response is shown as Waiting for response…​. Refresh the page after a few seconds to view the result (the lower portion of Figure 4). The value of the request Type column can be S or A, which is shell or agent command respectively.

Figure 4 illustrates that pstree -alp was passed to the agent. The pstree command shows a hierarchical representation of the process tree and is often useful to see what commands agents are running. Note that pstree is a child of ecbreakpoint and ecbreakpoint is a child of agent 0. The compilation and linking processes will probably be finished by the time you run pstree. You may see other processes, for example, processes launched in the background such as license managers, that may provide clues about what is happening.

Figure 4: ` pstree` command run from the breakpoint menu

You can also get session state information to send to Electric Cloud technical support. For contact information, see https://support.cloudbees.com/hc/en-us/categories/360002059512 .

Figure 5: Session state information

You can access ecbreakpoint from the command line or scripts using the curl command. CloudBees Build Acceleration ships with curl in the ecloud bin directory. Figure 3 shows the IP address and port for ecbreakpoint. Use those values in curl and make sure to use the appropriate escape characters for spaces, slashes, and so on.

Following is an example for running the ps command on the agent:

https://192.168.26.235:41184/exec.xml?command=%2Fbi2Fps%20-ef

Table 2 contains URLs you can use to access ecbreakpoint.

retry.xml

Rerun the command on a different agent.

continue.xml

Continue normal execution.

exec.xml

Execute the specified command. The command is one simple command passed with the command GET parameter.

agent.xml

Execute the specified agent command. The command is one simple command passed with the command GET parameter.

agentstat.xml

Show the current agent session state.

Table 2: ecbreakpoint access URLs

Labeling breakpoints

The makefile in Figure 1 contained one ecbreakpoint command. If you want to set multiple breakpoints in your build, you must use the ecbreakpoint --label command line option to distinguish them in the UI. Notice the --label option in the Makefile in Figure 6.

all: a b c @ echo Running $@ a: @ ecbreakpoint --label $@ b: @ ecbreakpoint --label $@ c: @ ecbreakpoint --label $@

Figure 6: Makefile with a label for each target

When the Makefile in Figure 6 runs, the Cluster Manager displays labels for each breakpoint (Figure 7).

breakpoint builddetails labels

Figure 7: Labeled breakpoints on the Build Details page

The label command may need more information than just the target name. For example, if a pattern rule was called multiple times from various submakes where the C files had the same names, this could result in a situation similar to the one in Figure 8 where the labels are the same.

breakpoint builddetails labels2

Figure 8: Breakpoint labels with the same name

You can make labels more distinct by specifying the label as --label="$(@) \_pwd", which results in the output in Figure 10.

%.c: @ touch $@ ` `@ ecbreakpoint --label=" $(@) \_`pwd`"

Figure 9: ` ecbreakpoint` label option augmented with the working directory

breakpoint builddetails labels3

Figure 10: Breakpoint labels with the working directory

Using agent-side breakpoints

You can use agent-side breakpoints when:

  • you do not know where in the makefile a problem occurs, but you want to stop the build based on some condition

  • you know, after the fact, where a problem occurs in a makefile, but the location keeps changing

You can use agent-side breakpoints to configure agents to run arbitrary commands when a particular condition is found. The arbitrary command can be ecbreakpoint, but it does not have to be. You can use the cmtool runAgentCmd program to configure agent-side breakpoints (Figure 1).

cmtool runAgentCmd 'session breakpoint set {trigger_code} {command_to_execute}' cmtool runAgentCmd 'session breakpoint get' cmtool runAgentCmd 'session breakpoint clear'

Figure 1: Session commands to manipulate agent-side breakpoints

Agent-side breakpoints enable agents to run a block of Tcl code before and after running the job command. The breakpoint consists of a block of trigger code and a command to run if the trigger code returns a non-zero return code. If the trigger block results in a non-zero exit code, the agent runs the command in the second block. The trigger code has access to the read-only breakpoint associative array that contains the entries in Table 1. Generally, the Tcl code you write will use regular expressions to match the stdout element of the breakpoint array. By default, eMake runs with the “merge streams” option so there is usually no reason to look in stderr.

Array Contents

breakpoint(command)

The command line

breakpoint(location)

“pre” or “post”, for before or after the command runs

breakpoint(exitcode)

Exit code of the command; -1 for “pre” checks

breakpoint(stdout)

Output of the command; empty for “pre” checks

breakpoint(stderr)

stderr of the command. Generally, mergestreams is enabled, so you can look at stdout.

Table 1: Breakpoint associative array

Figure 2 illustrates a breakpoint that triggers if the standard output of a command contains the text ` parse error` or ` undeclared identifier`.

# Set breakpoint using runAgentCmd cmtool --cm=support-lin1 runAgentCmd 'session breakpoint set { ` # examine stdout from breakpoint array to see if we have the error messages` ` # we are looking for` ` if { [ regexp {(parse error )|(No such file )} $breakpoint (stdout)] } {` ` # Send info about what we find to the agent logs` ` puts "Triggering breakpoint:"` ` puts " \t Command: $breakpoint(command)"` ` puts " \t Stdout: $breakpoint(stdout)"` ` return 1` ` }` ` return 0` } {ecbreakpoint --cm=support-lin1}' --agentName support-lin3-1

Figure 2: Agent breakpoint triggered by regex on stdout

When you set the breakpoint and run a makefile that emits the desired error strings (Figure 3), you see the breakpoints displayed on the Build Details page (Figure 4).

all: a b c @ echo Running $@ a: @ echo "good compilation" b: cat /this/is/a/really/long/string/that/doesnt/exist/on/the/computer c: @ echo "parse error"

Figure 3: Makefile to generate error

Using the agent name displayed in the Break Points panel (Figure 4), you can find the output from the agent breakpoint in the agent logs (/var/log/ecagent#.logs ) (Figure 5).

Figure 4: Cluster Manager UI for agent-side breakpoints

Triggering breakpoint: ` Command: /bin/sh -c echo "parse error"` ` Stdout: parse error`

Figure 5: /var/log/ecagent4.log

Figure 4 illustrates the empty Label column. You cannot use the ecbreakpoint --label argument in the Figure 2 breakpoint because the breakpoint is set before jobs are run. Remember that the agent-side breakpoint can run arbitrary commands. The breakpoint code in Figure 6 accesses the command-line string from the breakpoint array, constructs a shell script that calls ecbreakpoint with the --label you create, and then runs the shell script when the breakpoint triggers.

cmtool --cm=support-lin1 runAgentCmd 'session breakpoint set { ` # Examine stdout for error messages that we want to stop the build when encountered` ` if { [ regexp {(No such file )|(undeclared identifier)} $breakpoint (stdout)] } {` ` # cleanup the command line so that we can use it for a label.` ` # remove text that was prepended by the agent and change spaces to underscores` ` regsub {^/bin/sh -c } $::breakpoint (command) {} cmd` ` regsub -all {\s+} $cmd "" cmd` ` # If we have a long cmd, just use the first 20 and last 20 characters` ` # for the label` ` if {[ string length $cmd ] > 40 } {` ` regexp {^(.{ 20 }).*(.{ 20 })$} $cmd match cmd_a cmd_b` ` set cmd "${cmd_b}=${cmd_e}"` ` }` ` # Create a file so that we can write out the new ecbreakpoint command with the` ` # appropriate label for this job. By using [efs use], we can create a unique` ` # temp file for each agent` ` set f [ open /tmp/breakpoint[efs use].sh w]` ` puts $f "ecbreakpoint --cm=support-lin1 --label $cmd"` ` close $f` ` return 1` ` }` ` return 0` } [ list /bin/sh /tmp/breakpoint_[efs use].sh]'

Figure 6: Agent breakpoint that initiates an arbitrary program

When you run the makefile in Figure 3 with the new breakpoint code, more descriptive labels are printed in the Break Points panel (Figure 7).

breakpoint builddetails labels4

Figure 7: Breakpoint list on the Build Details page

Because the job path is not available to the agents, the example uses the command name as an identifier even though it is a poor substitute for the job path and might not always be unique. The jobid is available, but it does not correlate to the annotation file at the point in time when the breakpoint triggers because job details are not written to the annotation file until termination time.

Specifying a breakpoint through a file

You can also specify the breakpoint by passing in a file. Begin the command argument with file: <yourpath> and the file will be read from the path specified for the runAgentCmd request.

Example breakpoint contained in a file named test.txt :

session breakpoint set { ` if { [regexp {(parse error)|(No such file)} $breakpoint(stdout)] } {` ` puts "Triggering breakpoint:"` ` puts "\t Command: $breakpoint(command)"` ` return 1 } return 0 } {ecbreakpoint --cm=linbuild-cm}`

Example of passing in the breakpoint contained in test.txt :

cmtool runAgentCmd "file:C:\tmp\test.txt" --agentId 20

Troubleshooting breakpoints

After you create an agent-side breakpoint, it might not work as expected or provide any output. following are possible problems and solutions:

Breakpoint does not trigger

If the breakpoint does not trigger when you think it should, look in the agent log files. If there are Tcl errors with your breakpoint, you may see something like the following:

[228/0] 0.081472/105997992.238145 SYSTEM_LOG: INFO Error evaluating breakpoint condition: can't read "agentId": no such variableTcl errors will prevent the breakpoint from running.

Breakpoint regex is wrong

Be sure that the text you are looking for actually appears in the stdout. If the text is in the output, open the tkcon console (in the ecloud bin directory) and test your regex code. Modify the regex until it works.

Output is not as expected

Output from agent-side breakpoint 'puts' statements is written to agent logs and not the build’s stdout. If the agent breakpoint stops the build with ecbreakpoint, you can use the Break Points panel to determine which agent log to examine. However, if your breakpoint calls a tool such as strace, you may have to examine all agent logs. On Linux, one useful hint to remember is that Gnu ` tail(1)` can tail multiple files at the same time ( tail -f /var/log/ecagent?.log ).

Use your editor to run breakpoints

When creating agent breakpoints, it is useful to have an open editor window with the cmtool runAgentCommand set breakpoint command that you are modifying. When working with the breakpoint, you can execute it from within the editor by selecting the code in Vim’s visual mode and then using the 'ex' command line to feed it into the shell for execution. You do not need to clear the breakpoint before setting a new one because setting a breakpoint overwrites the current one. You can also cut-and-paste the breakpoint code into a command window.

In the .vimrc, configure the makeprg variable to call eMake instead of GNU Make; type ` :make` from the ex command line to invoke the makefile.

set makeprg=/opt/ecloud/i686_Linux/bin/emake\ --emake-cm=support-lin1

In emacs, you can feed text to the shell using 'M-|'. Select the region of the buffer you want to send to the shell, and then type M-|.