Sending email notification when a Jenkins failover occurs

Email notifications can be very useful when your highly available Jenkins controller(1) fails-over from one node to the other. Such notification can help to review the list of aborted builds or to look at the failed node.

To implement email notification on Jenkins controller failover, please follow the steps:

Read the "CloudBees Jenkins Enterprise HA monitor tool" section of the "[CloudBees Jenkins Enterprise » High Availability][CloudBees Jenkins Enterprise _ High Availability]`" user guide
On each Jenkins controller of the highly available cluster
- Download the "jenkins-ha-monitor-1.2-jar-with-dependencies.jar" jar from the [Jenkins HA monitor (active/passive) download section][] (where “1.2” is the version of the tool, you should pick the latest one)
- Install a shell notification script "send-notification-email-on-jenkins-controller-promotion.sh" similar tomail -s "[Jenkins HA failover] controller hostname has been promoted`"
  [jenkins-admin@example.com][jenkins-admin_example.com] «< "Jenkins controller hostname has been promoted at `date`"
- Launch the process
  java -jar jenkins-ha-monitor-4.3-jar-with-dependencies.jar \ -home /mnt/nfs/jenkins-home/ \ -host-promotion /path/to/send-notification-email-on-jenkins-controller-promotion.sh
  - The linux user used to start this process must have read & write access to the shared JENKINS_HOME
  - -home is the path to the JENKINS_HOME
  - -host-promotion is the path of the script invoked when the node is promoted

Sample of logs of the HA Monitor Tool on the Jenkins controller node that gets promoted:

Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.singleton.HASingleton$3 viewAccepted
INFO: Cluster membership has changed to: \[ip-10-0-0-12-45063 (309 bytes)|7\] \[ip-10-0-0-12-45063 (309 bytes), ip-10-0-0-11-33579 (272 bytes), ip-10-0-0-11-46466 (272 bytes), ip-10-0-0-12-54079 (272 bytes)\]
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.singleton.HASingleton$3 viewAccepted
INFO: New primary node is JenkinsClusterMemberIdentity\[member=ip-10-0-0-12-45063 (309 bytes),weight=0,min=0\]
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.monitor.GraveDigger$1 run
INFO: Processing the fail over from JenkinsClusterMemberIdentity\[member=ip-10-0-0-11-8467 (309 bytes),weight=0,min=0\] to JenkinsClusterMemberIdentity\[member=ip-10-0-0-12-45063 (309 bytes),weight=0,min=0\]
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.monitor.GraveDigger promoteHost
WARNING: This host is now hosting the primary node
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.monitor.GraveDigger run
INFO: Executing /home/ec2-user/send-notification-email-on-jenkins-controller-promotion.sh
Tue Aug  5 18:22:20 UTC 2014 host promotion
Aug 05, 2014 6:22:20 PM com.cloudbees.jenkins.ha.monitor.GraveDigger run
INFO: Executed /home/ec2-user/send-notification-email-on-jenkins-controller-promotion.sh: exit code=0

Notes:

The jenkins-ha-monitor allows to attach scripts to two events: “on-promotion” when the controller gets promoted as the active controller and “on-demotion” when the controller becomes passive
In very specific cases, the “on-promotion” and “on-demotion” events can be invoked on a jenkins controller that is respectively already active or already standing-by. For example, the on-promotion event is triggered on the active node if the monitor-tool restarts

(1) High Availability (active/passive) is a feature of CloudBees Jenkins Enterprise by CloudBees.

CloudBees Jenkins Enterprise _ High Availability (active/passive)