Validated Merge plugin
In a large software development project, where many developers work on the same code, keeping the repository stable is a challenge. Because of sheer number of commits that land on the repository, even if the chance of accidental regression per commit is small, they compound and make the repository unstable, blocking other developers.
A traditional implementation of Jenkins, where it tests the tip of the repository, doesn’t help with this situation as regressions can be only discovered after they land on the tip of the repository.
Therefore often the "solution" to the problem is to have every developer rigorously run tests before changes are pushed to the repository, thereby wasting precious human time on test executions, which ideally should be running on the server.
The validated merge feature for Git is a plugin that solves this problem by allowing developers to push commits (called "proposed changes") to Jenkins, specifying which branch it is intended for (called "upstream ref", for example "master".) Jenkins will then build the proposed changes, verify that it is of a good quality, and then push that change to the upstream repository.
In this way, developers can now engage in the fire-and-forget style development. Once he is reasonably happy with the changes he made, he can push this into Jenkins and then move on to the next task, without waiting for a lengthy test cycle to complete. If the proposed changes are actually good, it’ll automatically land in the main repository. If not, he will get a notification.
In this way, the validated merge feature allows you to offload more work from developers and their laptops to Jenkins and its agents.
This feature is contained in the
git-validated-merge plugin. It is
enabled by default for CloudBees CI, but if you don’t
find it, go install this plugin. Go to your CI build/test job
configuration page and check Enable Git validated merge support. Once
this option is selected, you’ll see the Git Repository for Validated
Merge item appear in the menu list of the job page:
Depending on how you configure your security setting, you might see a warning in this page asking you to fix the port number of the Jenkins SSHD service. If you see it, go to the system configuration page and set the SSHD port under the "SSH Server" section, as shown below.
Then, make sure that Jenkins can push to your upstream Git repository. See Pushing to the upstream repository for more discussion.
Your server is now ready for the validated merge.
Now, on a developer laptop where you’d like to use the validated merge feature, register this Jenkins job as the remote repository. The exact URL of the Git repository in the Jenkins job can be obtained by the "Git Repository for Validated Merge" page on the job top page. You might see an HTTP URL instead of SSH URL, if you haven’t enabled the security setting.
$ git remote add jenkins ssh://example.com:2222/foo
Instead of registering this as a separate repository, you can also set Jenkins as the push URL of your upstream repository. In this way, your push will always go through Jenkins automatically, without you typing anything differently from command line. In comparison, the previous approach would require that you consciously select Jenkins as the target of each push.
$ git remote set-url --push origin ssh://example.com:2222/foo
This needs to be repeated for every workspace, but now your client is also ready for the validated merge.
Before we go discussing how you actually do a validated merge, let’s briefly talk about the mental picture of how this whole thing works.
With the validated merge, your Jenkins job acts as an intermediate repository between your workspace and your central/upstream repository. Instead of pushing changes directory to the upstream repository, you’ll push changes to the Jenkins job. We’ll refer to this intermediate repository as the gate repository.
The gate repository implemented by Jenkins is a little bit magical. When
you push a ref to the gate repository (for example via
git push jenkins master), it’ll lie to you that the push was
successful and the ref was moved, to make your client happy, but it
actually hasn’t done so. Instead, it’ll just remember the ref that you
wanted to push to (
master in this case), and the actual commit you
Jenkins then schedules a build for your newly pushed commit. In this
build, it’ll check out the ref from the upstream you wanted to push the
changes into (
origin/master in this case), then merge your changes
into it. The build will then run as usual, and if the build was
successful, this result of the merge that was just built and tested will
be pushed to the upstream repository, becoming the new head of the
branch. This process happens completely automatically.
Unlike Gerrit+Jenkins integration, in which each commit is separately built and tested, the validated merge in Jenkins only checks the tip of the ref. So you can split your change into several logically separated pieces, without paying the penalty of more build time.
The gate repository hosts additional tags that can be useful. For
example, every submitted commit is accessible via
is a build number, and every commit that was actually built is
build/N is a result of a merge of the
changes/N. This allows one developer to look into a
failure found in a commit pushed by another developer and help him fix
Let’s get back to the developer laptop and make a change to the code base. When you are done, make a commit. To see the effect of validation, we’ll have this change break the build.
# just to be clear that you are working on the master branch $ git checkout master # touch some file and commit $ touch some.c $ echo '#!/bin/sh\nfalse' > run.sh $ git add some.c run.sh $ git commit -am "adding some file" [master 363f629] adding some file ...
Now the developer is happy with the changes. He thinks he’s done; he hasn’t run a full build/test cycle, but he thinks he made all the necessary changes for fixing a bug.
So let’s send this change to Jenkins to make sure they actually do pass all the tests.
$ git push jenkins master ... remote: Created refs/heads/validated-merge-for/master/20120415-095424 remote: at 363f629 for refs/heads/master To ssh://example.com:2022/foo/repo.git * [new branch] master -> master
Switch to the Jenkins web UI, and you’ll see a build scheduled for this
new push. If you look at its console output, you’ll see that it’s
checking out the upstream repo and then merging the change that was just
submitted, and the result (
8ded3bb) is getting built.
Using strategy: Build commits submitted for validated merge Last Built Revision: Revision 4ced883 (master) Fetching changes from 1 remote Git repository Fetching upstream changes from /my/upstream.git ... Merging refs/tags/changes/47 Commencing build of Revision 363f629 (master) [workspace] $ /bin/sh -xe /tmp/hudson9087427762386246304.sh + ./run.sh Build step 'Execute shell' marked build as failure Finished: FAILURE
The build has failed, and so the push to the upstream repository didn’t
happen. Let’s verify this from the perspective of Alice, another
developer working on this code base. Pretend that you are Alice, and
check out the workspace into another location. You’ll see that the
master branch of Alice doesn’t contain the problematic
To make this tutorial interesting, let’s make some changes as Alice in the master. In a big repository, it is normal for changes to overlap like this.
$ echo hello >> alice $ git add hello $ git commit -am "edit by alice" [master 834782e] edit by alice $ git push origin master ... To /my/upstream.git 458ea81..834782e master -> master
At this point, the master branch in the upstream is
834782e as pushed
by Alice, while our initial change
363f629 was rejected by Jenkins.
We’ll see how this will resolve itself in the end.
Now, switch back to the original workspace, and let’s pretend that we found out that our initial commit was rejected because it didn’t pass the build. Now, make a correction to the commit, and push that to Jenkins again.
$ echo '#!/bin/sh\ntrue' > run.sh $ git commit -am "fixed a broken build" [master f664265] fixed a broken build $ git push jenkins master remote: Created refs/heads/validated-merge-for/master/20120415-105638 remote: at f664265 for refs/heads/master To ssh://example.com:2022/foo/repo.git * [new branch] master -> master
When you make these corrections, you have the choice of amending the commit or doing a separate commit, and Jenkins can handle both correctly. In a silly regression like this, amending a commit is probably better, but for more intricate problems, leaving the record of a failure and its resolution could be useful.
Also note that we did not pull the changes Alice just made.
Now switch to the Jenkins UI once again, and see the console output of a newly scheduled build.
Using strategy: Build commits submitted for validated merge Fetching changes from 1 remote Git repository Fetching upstream changes from /my/upstream.git ... Merging refs/tags/changes/48 Commencing build of Revision f884547 (master) Checking out Revision f884547 (master) [workspace] $ /bin/sh -xe /tmp/hudson1267891670488600118.sh + ./run.sh Pushing the result to origin Finished: SUCCESS
As you can see, at the very end Jenkins pushed the changes up to the upstream repository because the build was successful.
Go back to the workspace and pull from the origin to verify that the changes did make it into the upstream:
$ git pull origin From /my/upstream 458ea81..f884547 master -> origin/master Updating f664265..f884547 Fast-forward ...
This updates your local workspace by pulling in all the changes made in the trunk.
To more clearly see what has happened, run the
git log to see the commit
graph. We first committed
which didn’t work. Then Alice committed
834782e, which was pushed
directly to the master. We then committed
f664265 as a follow-up fix,
which was merged with
834782e (then tip of the
master branch) to
f884547 (see the above console output for build #48 and you
see that it actually tested this), which then became the tip of the
$ git log --decorate --graph --date-order * commit f884547 (HEAD, origin/master, master) |\ Merge: 834782e f664265 | | Author: Jenkins | | | | Merge commit 'refs/tags/changes/48' | | | * commit f664265 (jenkins/master) | | Author: Kohsuke Kawaguchi | | | | fixed a broken build | | * | commit 834782e | | Author: Alice | | | | edit by alice | | | * commit 363f629 |/ Author: Kohsuke Kawaguchi | | adding some file | * commit 458ea81
So now we completed a successful validated merge.
This feature defines two permissions that can be used to control access to the repository.
This permission controls whether the user can submit a change for a validated merge. This permission is implied by the build permission of the job, meaning those who can trigger a build can submit changes by default.
This permission controls whether the user can retrieve commits from the gate repository. This permission allows users to retrieve other people’s submissions. This permission is implied by the push permission, meaning those who can push changes into the gate repository will also automatically be able to retrieve changes.
This feature provides two transports to access the gate repository. One is the smart HTTP protocol, and the other is the SSH protocol.
If your job has the URL
http://myjenkins/job/foo/job/bar/, then the
Git repository via HTTP is available at
http://myjenkins/job/foo/bar/repo.git. If your Jenkins is configured
without security, this URL can be used as is. Otherwise log in to
Jenkins and visit this page in a web browser to get a personal access
URL. That URL allows Git to access the repository under your credentials
without sending a password; keep this URL secret.
The SSH access to this same repository is available at
NNNN is the port number of the
Jenkins SSHD service. If your Jenkins is configured with security,
you’ll need to register your public key with Jenkins to authenticate the
The HTTP protocol access is available all the time, and the SSH access is available if and only if the Jenkins SSHD service is available. The "Git repository for validated merge" UI will by default offer the SSH URL when SSHD is running, falling back to the HTTP URL; but an administrator can override this preference (as well as the SSHD service) in the Jenkins global configuration page.
In many cases the real upstream repository will also have access control and not permit anonymous pushes. To permit Jenkins to push your validated commits (along with associated merge commits) to the upstream repository, click the Advanced button in the job’s configuration page and select Credentials for pushing upstream. This could be a HTTP(S) username and password combination, or an SSH private key. You can also pick <automatic by pushing user> in which case the set of credentials associated with the user pushing to the gate repository will be searched for those matching the upstream URL.
When you enable a validated merge support for a job, the same job can still be used for a regular CI build, where it builds the tips of the upstream repository. Builds triggered outside a push to a gate repository (such as someone clicking "Build Now", scheduled execution, trigger via polling, and trigger from other builds) will result in the regular build behavior.
When a validated merge build succeeds, Jenkins will push the merge commit to the upstream repository, but if someone else has pushed new commits to the same branch while Jenkins was validating the submitted changes, this push will fail. The Git jargon for this is that the push is not a fast-forward. Causing a build to fail because of this is not necessarily desirable, because there actually wasn’t any problem in the submitted changes. In the "advanced" section of the "Enable Git validated merge support" feature in the job configuration screen, you can tell Jenkins how to deal with this situation.
- Merge and push
Given that the submitted changes were OK, Jenkins will perform another merge between the commit that was just validated and the commit that’s currently in the upstream repository, then push the new merge commit to the upstream. If a merge fails, the build will be marked as a failure. This is basically assuming that there are no undesirable interaction between the changes that were submitted to Jenkins and the changes that were pushed to the upstream repository. This requires less computing cycles, but it has a potential risk of broken builds in the branch tip. This is generally a desirable option if your validated merge takes a significant amount of time.
- Redo a validated merge
Start a new validated merge all over again, with the newly discovered commit in the upstream repository. The current build gets marked as aborted, then a new one will be started with the same submitted changes. This requires additional computing cycles, but it will guarantee that no untested commits ever land on the upstream repository. This is generally a desirable option if your validated merge completes quickly.
- Fail the build if push fails
Cause a build to fail. This is essentially asking the submitter of the changes to deal with the situation. He can then choose to resubmit the changes, or push straight into the upstream. This is a desirable option if none of the other options suit your needs.
This behavior is extensible. Other plugins can implement custom behaviors.
The gate repository holds tags that follow a certain naming conventions:
This tag indicates the validated merge commit that was submitted by the user and built in the build #NNN.
This tag indicates the commit that was actually built and tested in the build #NNN. This is the result of a merge between the then-tip of the upstream ref, and
changes/NNN. (Therefore if this merge was a fast-forward, this tag points to the same commit as
changes/NNNtags are more easily memorable but they are only available once the build has started. Therefore, when a proposed change is pushed to the gate repository, Jenkins assigns a tag that follows this longer format. The
UPSTREAMportion refers to the upstream ref that this change is meant for (such as 'master', but this can include '/' in it, such as 'feature/foo'), and
YYYYMMDD-HHMMSSis the timestamp of the submission. If multiple changes are pushed within the same second, the additional
Xis a number) is appended to create unique tags.
These tags are useful for developers to collaborate on changes that are being validated or rejected. For example, if a developer submitted a change and it was rejected, another developer can fetch the change, make an additional commit, then submit it to integrate that to the upstream.
Aside from these tags, the gate repository also contains any tags that are set in the upstream repository, as well as tags that are created in the Jenkins workspace. For example, the Git plugin add its own tag for every build.
Since the gate repository tends to host a large number of tags, you
normally don’t want to fetch every single tag in it. To fetch a specific
tag and that alone, run Git like
git fetch -n tag changes/123 where
'-n' prevents Git from fetching tags
automatically and "tag changes/123" tells Git to retrieve the specific
The gate repository also contains branches that map to the permalinks in
the job in the form
ID refers to the identifier
of the permalink, such as
Other plugins often define additional permalinks, for example the
promoted builds plugin.
These tags can be useful beyond the validated merge use case. For example, when a build is promoted, you can start another job which checks out the branch that corresponds to the promotion and then deploys to the staging server.