Validated Merge plugin

In a large software development project, where many developers work on the same code, keeping the repository stable is a challenge. Because of sheer number of commits that land on the repository, even if the chance of accidental regression per commit is small, they compound and make the repository unstable, blocking other developers.

A traditional implementation of Jenkins, where it tests the tip of the repository, doesn’t help with this situation as regressions can be only discovered after they land on the tip of the repository.

Therefore often the "solution" to the problem is to have every developer rigorously run tests before changes are pushed to the repository, thereby wasting precious human time on test executions, which ideally should be running on the server.

The validated merge feature for Git is a plugin that solves this problem by allowing developers to push commits (called "proposed changes") to Jenkins, specifying which branch it is intended for (called "upstream ref", for example "master".) Jenkins will then build the proposed changes, verify that it is of a good quality, and then push that change to the upstream repository.

In this way, developers can now engage in the fire-and-forget style development. Once he is reasonably happy with the changes he made, he can push this into Jenkins and then move on to the next task, without waiting for a lengthy test cycle to complete. If the proposed changes are actually good, it’ll automatically land in the main repository. If not, he will get a notification.

In this way, the validated merge feature allows you to offload more work from developers and their laptops to Jenkins and its agents.

Tutorial

Server setup

This feature is contained in the git-validated-merge plugin. It is enabled by default for CloudBees CI, but if you don’t find it, go install this plugin. Go to your CI build/test job configuration page and check Enable Git validated merge support. Once this option is selected, you’ll see the Git Repository for Validated Merge item appear in the menu list of the job page:

repo page
Figure 1. The "Git Repository for Validated Merge" screen

Depending on how you configure your security setting, you might see a warning in this page asking you to fix the port number of the Jenkins SSHD service. If you see it, go to the system configuration page and set the SSHD port under the "SSH Server" section, as shown below.

sshd port
Figure 2. Configuring the SSHD port

Then, make sure that Jenkins can push to your upstream Git repository. See Pushing to the upstream repository for more discussion.

Your server is now ready for the validated merge.

Client setup

Now, on a developer laptop where you’d like to use the validated merge feature, register this Jenkins job as the remote repository. The exact URL of the Git repository in the Jenkins job can be obtained by the "Git Repository for Validated Merge" page on the job top page. You might see an HTTP URL instead of SSH URL, if you haven’t enabled the security setting.

$ git remote add jenkins ssh://example.com:2222/foo

Instead of registering this as a separate repository, you can also set Jenkins as the push URL of your upstream repository. In this way, your push will always go through Jenkins automatically, without you typing anything differently from command line. In comparison, the previous approach would require that you consciously select Jenkins as the target of each push.

$ git remote set-url --push origin ssh://example.com:2222/foo

This needs to be repeated for every workspace, but now your client is also ready for the validated merge.

Mental picture of the validated merge

Before we go discussing how you actually do a validated merge, let’s briefly talk about the mental picture of how this whole thing works.

With the validated merge, your Jenkins job acts as an intermediate repository between your workspace and your central/upstream repository. Instead of pushing changes directory to the upstream repository, you’ll push changes to the Jenkins job. We’ll refer to this intermediate repository as the gate repository.

The gate repository implemented by Jenkins is a little bit magical. When you push a ref to the gate repository (for example via git push jenkins master), it’ll lie to you that the push was successful and the ref was moved, to make your client happy, but it actually hasn’t done so. Instead, it’ll just remember the ref that you wanted to push to (master in this case), and the actual commit you pushed.

tres repos
Figure 3. Repository model

Jenkins then schedules a build for your newly pushed commit. In this build, it’ll check out the ref from the upstream you wanted to push the changes into (origin/master in this case), then merge your changes into it. The build will then run as usual, and if the build was successful, this result of the merge that was just built and tested will be pushed to the upstream repository, becoming the new head of the branch. This process happens completely automatically.

Unlike Gerrit+Jenkins integration, in which each commit is separately built and tested, the validated merge in Jenkins only checks the tip of the ref. So you can split your change into several logically separated pieces, without paying the penalty of more build time.

The gate repository hosts additional tags that can be useful. For example, every submitted commit is accessible via changes/N where N is a build number, and every commit that was actually built is accessible via build/N (build/N is a result of a merge of the upstream and changes/N. This allows one developer to look into a failure found in a commit pushed by another developer and help him fix the problem.

Sending changes and have them validated

Let’s get back to the developer laptop and make a change to the code base. When you are done, make a commit. To see the effect of validation, we’ll have this change break the build.

# just to be clear that you are working on the master branch
$ git checkout master

# touch some file and commit
$ touch some.c
$ echo '#!/bin/sh\nfalse' > run.sh
$ git add some.c run.sh
$ git commit -am "adding some file"
[master 363f629] adding some file
 ...

Now the developer is happy with the changes. He thinks he’s done; he hasn’t run a full build/test cycle, but he thinks he made all the necessary changes for fixing a bug.

So let’s send this change to Jenkins to make sure they actually do pass all the tests.

$ git push jenkins master
 ...
remote: Created refs/heads/validated-merge-for/master/20120415-095424
remote:  at 363f629 for refs/heads/master
To ssh://example.com:2022/foo/repo.git
 * [new branch]      master -> master

Switch to the Jenkins web UI, and you’ll see a build scheduled for this new push. If you look at its console output, you’ll see that it’s checking out the upstream repo and then merging the change that was just submitted, and the result (8ded3bb) is getting built.

Using strategy: Build commits submitted for validated merge
Last Built Revision: Revision 4ced883 (master)
Fetching changes from 1 remote Git repository
Fetching upstream changes from /my/upstream.git
  ...
Merging refs/tags/changes/47
Commencing build of Revision 363f629 (master)
[workspace] $ /bin/sh -xe /tmp/hudson9087427762386246304.sh
+ ./run.sh
Build step 'Execute shell' marked build as failure
Finished: FAILURE

The build has failed, and so the push to the upstream repository didn’t happen. Let’s verify this from the perspective of Alice, another developer working on this code base. Pretend that you are Alice, and check out the workspace into another location. You’ll see that the master branch of Alice doesn’t contain the problematic 363f629.

To make this tutorial interesting, let’s make some changes as Alice in the master. In a big repository, it is normal for changes to overlap like this.

$ echo hello >> alice
$ git add hello
$ git commit -am "edit by alice"
[master 834782e] edit by alice
$ git push origin master
...
To /my/upstream.git
   458ea81..834782e  master -> master

At this point, the master branch in the upstream is 834782e as pushed by Alice, while our initial change 363f629 was rejected by Jenkins. We’ll see how this will resolve itself in the end.

Now, switch back to the original workspace, and let’s pretend that we found out that our initial commit was rejected because it didn’t pass the build. Now, make a correction to the commit, and push that to Jenkins again.

$ echo '#!/bin/sh\ntrue' > run.sh
$ git commit -am "fixed a broken build"
[master f664265] fixed a broken build
$ git push jenkins master
remote: Created refs/heads/validated-merge-for/master/20120415-105638
remote:   at f664265 for refs/heads/master
To ssh://example.com:2022/foo/repo.git
 * [new branch]      master -> master

When you make these corrections, you have the choice of amending the commit or doing a separate commit, and Jenkins can handle both correctly. In a silly regression like this, amending a commit is probably better, but for more intricate problems, leaving the record of a failure and its resolution could be useful.

Also note that we did not pull the changes Alice just made.

Now switch to the Jenkins UI once again, and see the console output of a newly scheduled build.

Using strategy: Build commits submitted for validated merge
Fetching changes from 1 remote Git repository
Fetching upstream changes from /my/upstream.git
  ...
Merging refs/tags/changes/48
Commencing build of Revision f884547 (master)
Checking out Revision f884547 (master)
[workspace] $ /bin/sh -xe /tmp/hudson1267891670488600118.sh
+ ./run.sh
Pushing the result to origin
Finished: SUCCESS

As you can see, at the very end Jenkins pushed the changes up to the upstream repository because the build was successful.

Go back to the workspace and pull from the origin to verify that the changes did make it into the upstream:

$ git pull origin
From /my/upstream
   458ea81..f884547  master     -> origin/master
Updating f664265..f884547
Fast-forward
 ...

This updates your local workspace by pulling in all the changes made in the trunk.

To more clearly see what has happened, run the git log to see the commit graph. We first committed 363f629, which didn’t work. Then Alice committed 834782e, which was pushed directly to the master. We then committed f664265 as a follow-up fix, which was merged with 834782e (then tip of the master branch) to produce f884547 (see the above console output for build #48 and you see that it actually tested this), which then became the tip of the master branch.

$ git log --decorate --graph --date-order
*   commit f884547 (HEAD, origin/master, master)
|\  Merge: 834782e f664265
| | Author: Jenkins
| |
| |     Merge commit 'refs/tags/changes/48'
| |
| * commit f664265 (jenkins/master)
| | Author: Kohsuke Kawaguchi
| |
| |     fixed a broken build
| |
* | commit 834782e
| | Author: Alice
| |
| |     edit by alice
| |
| * commit 363f629
|/  Author: Kohsuke Kawaguchi
|
|       adding some file
|
*   commit 458ea81

So now we completed a successful validated merge.

Access Control

Permissions

This feature defines two permissions that can be used to control access to the repository.

Push

This permission controls whether the user can submit a change for a validated merge. This permission is implied by the build permission of the job, meaning those who can trigger a build can submit changes by default.

Pull

This permission controls whether the user can retrieve commits from the gate repository. This permission allows users to retrieve other people’s submissions. This permission is implied by the push permission, meaning those who can push changes into the gate repository will also automatically be able to retrieve changes.

Accessing the gate repository

This feature provides two transports to access the gate repository. One is the smart HTTP protocol, and the other is the SSH protocol.

If your job has the URL http://myjenkins/job/foo/job/bar/, then the Git repository via HTTP is available at http://myjenkins/job/foo/bar/repo.git. If your Jenkins is configured without security, this URL can be used as is. Otherwise log in to Jenkins and visit this page in a web browser to get a personal access URL. That URL allows Git to access the repository under your credentials without sending a password; keep this URL secret.

The SSH access to this same repository is available at ssh://myjenkins:NNNN/foo/bar.git where NNNN is the port number of the Jenkins SSHD service. If your Jenkins is configured with security, you’ll need to register your public key with Jenkins to authenticate the SSH access.

The HTTP protocol access is available all the time, and the SSH access is available if and only if the Jenkins SSHD service is available. The "Git repository for validated merge" UI will by default offer the SSH URL when SSHD is running, falling back to the HTTP URL; but an administrator can override this preference (as well as the SSHD service) in the Jenkins global configuration page.

Pushing to the upstream repository

In many cases the real upstream repository will also have access control and not permit anonymous pushes. To permit Jenkins to push your validated commits (along with associated merge commits) to the upstream repository, click the Advanced button in the job’s configuration page and select Credentials for pushing upstream. This could be a HTTP(S) username and password combination, or an SSH private key. You can also pick <automatic by pushing user> in which case the set of credentials associated with the user pushing to the gate repository will be searched for those matching the upstream URL.

Reusing a job between CI and validated merge

When you enable a validated merge support for a job, the same job can still be used for a regular CI build, where it builds the tips of the upstream repository. Builds triggered outside a push to a gate repository (such as someone clicking "Build Now", scheduled execution, trigger via polling, and trigger from other builds) will result in the regular build behavior.

Dealing with post-build push failure

When a validated merge build succeeds, Jenkins will push the merge commit to the upstream repository, but if someone else has pushed new commits to the same branch while Jenkins was validating the submitted changes, this push will fail. The Git jargon for this is that the push is not a fast-forward. Causing a build to fail because of this is not necessarily desirable, because there actually wasn’t any problem in the submitted changes. In the "advanced" section of the "Enable Git validated merge support" feature in the job configuration screen, you can tell Jenkins how to deal with this situation.

Merge and push

Given that the submitted changes were OK, Jenkins will perform another merge between the commit that was just validated and the commit that’s currently in the upstream repository, then push the new merge commit to the upstream. If a merge fails, the build will be marked as a failure. This is basically assuming that there are no undesirable interaction between the changes that were submitted to Jenkins and the changes that were pushed to the upstream repository. This requires less computing cycles, but it has a potential risk of broken builds in the branch tip. This is generally a desirable option if your validated merge takes a significant amount of time.

Redo a validated merge

Start a new validated merge all over again, with the newly discovered commit in the upstream repository. The current build gets marked as aborted, then a new one will be started with the same submitted changes. This requires additional computing cycles, but it will guarantee that no untested commits ever land on the upstream repository. This is generally a desirable option if your validated merge completes quickly.

Fail the build if push fails

Cause a build to fail. This is essentially asking the submitter of the changes to deal with the situation. He can then choose to resubmit the changes, or push straight into the upstream. This is a desirable option if none of the other options suit your needs.

This behavior is extensible. Other plugins can implement custom behaviors.

Refs in the gate repository

Tags

The gate repository holds tags that follow a certain naming conventions:

changes/NNN

This tag indicates the validated merge commit that was submitted by the user and built in the build #NNN.

builds/NNN

This tag indicates the commit that was actually built and tested in the build #NNN. This is the result of a merge between the then-tip of the upstream ref, and changes/NNN. (Therefore if this merge was a fast-forward, this tag points to the same commit as changes/NNN.)

validated-merge-for/UPSTREAM/YYYYMMDD-HHMMSS[.X]

The changes/NNN tags are more easily memorable but they are only available once the build has started. Therefore, when a proposed change is pushed to the gate repository, Jenkins assigns a tag that follows this longer format. The UPSTREAM portion refers to the upstream ref that this change is meant for (such as 'master', but this can include '/' in it, such as 'feature/foo'), and YYYYMMDD-HHMMSS is the timestamp of the submission. If multiple changes are pushed within the same second, the additional .X (where X is a number) is appended to create unique tags.

These tags are useful for developers to collaborate on changes that are being validated or rejected. For example, if a developer submitted a change and it was rejected, another developer can fetch the change, make an additional commit, then submit it to integrate that to the upstream.

Aside from these tags, the gate repository also contains any tags that are set in the upstream repository, as well as tags that are created in the Jenkins workspace. For example, the Git plugin add its own tag for every build.

Since the gate repository tends to host a large number of tags, you normally don’t want to fetch every single tag in it. To fetch a specific tag and that alone, run Git like git fetch -n tag changes/123 where '-n' prevents Git from fetching tags automatically and "tag changes/123" tells Git to retrieve the specific tag.

Branches

The gate repository also contains branches that map to the permalinks in the job in the form permalink/ID where ID refers to the identifier of the permalink, such as lastSuccessfulBuild or lastFailedBuild. Other plugins often define additional permalinks, for example the promoted builds plugin.

These tags can be useful beyond the validated merge use case. For example, when a build is promoted, you can start another job which checks out the branch that corresponds to the promotion and then deploys to the staging server.