Installing ZooKeeper

5 minute read

Use Apache Zookeeper Version 3.4.6 or later to maintain and synchronize group services in a clustered CloudBees Flow configuration. For more information, go to the link:https://zookeeper.apache.org/Apache ZooKeeper website .

For your convenience, ZooKeeper 3.4.6 is bundled in your CloudBees Flow installation here: <install dir>/utils.

To install ZooKeeper:

  1. Use either the bundled ZooKeeper or download ZooKeeper from the link:https://zookeeper.apache.org/releases.htmlZooKeeper website .

  2. Extract and install the files into an appropriate location. For example, ` /opt/zookeeper-<release_version>`

You must install ZooKeeper on an odd number of machines. The number of machines will determine if you install ZooKeeper in standalone mode (for one machine) or in replicated mode (for three or five machines).
  1. Create a zoo.cfg configuration file for each machine with an instance of ZooKeeper. For example, zookeeper-<release_version>/conf/zoo.cfg.

    • For standalone mode on a single ZooKeeper machine, the file has these values:

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
  • For replicated mode across multiple ZooKeeper machines, the zoo.cfg file on each server should have these values:

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=<ZooKeeper_hostname_1>:2888:3888
server.2=<ZooKeeper_hostname_2>:2888:3888
server.3=<ZooKeeper_hostname_3>:2888:3888
server.4=<ZooKeeper_hostname_4>:2888:3888
server.5=<ZooKeeper_hostname_5>:2888:3888

Where < ZooKeeper_hostname_1 > through < ZooKeeper_hostname_5 > are the hostnames of the servers for the ZooKeeper service.

The file has only three server value lines for a three-ZooKeeper configuration.

Create /var/lib/zookeeper/myid files on each ZooKeeper server, with each containing a single ASCII digit: 1 for the first server, 2 for the second server, and so on, corresponding to their server. <digit> values in the zoo.cfg files

  1. Create /var/lib/zookeeper/myid files on each ZooKeeper server, with each containing a single ASCII digit as follows:

    • 1 for the first server,

    • 2 for the second server

    • Up to 5, corresponding to the appropriate server. <digit> value in the zoo.cfg file.

  2. Start ZooKeeper by entering the following command:

Running ZooKeeper as a Service on Linux

Install ZooKeeper on machines other than those running CloudBees Flow. ZooKeeper must not be run on the same machines as those running the CloudBees Flow servers.

To run ZooKeeper as a service (running as the root user), follow these steps:

  1. For each machine on which you want to install ZooKeeper, navigate to the zookeeper-wrapper.zip file in the CloudBees Flow <install dir>/utils directory.

  2. Copy or move the zookeeper-wrapper.zip file to each of the machines that you want to use for ZooKeeper.

  3. Extract zookeeper-wrapper.zip to a directory of your choice.

    For example, ` /opt/zookeeper-<release_version>`.

  4. Using a text editor, open the wrapper.conf file in the extracted zookeeper-wrapper/conf directory.

    For example, /opt/zookeeper-<release_version>/zookeeper-wrapper/conf.

  5. Add the path to the ZooKeeper home directory. For example:

    # Path to unpacked zookeeper

    set.default.ZOOKEEPER_HOME=/opt/zookeeper-<release_version>

  6. Add the path to the JRE. For example:

    set.default.JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
  7. Edit /opt/zookeeper-<release_version>/zookeeper-wrapper/bin/zookeeper to change:

    WRAPPER_ROOT="../"

    to

    WRAPPER_ROOT="/opt/zookeeper-<release_version>/zookeeper-wrapper"

    For example:

    WRAPPER_ROOT="/opt/zookeeper-3.4.10/zookeeper-wrapper"
  8. Verify that you can start the ZooKeeper service by entering:

    sudo /opt/zookeeper-<release_version>/zookeeper-wrapper/bin/zookeeper start
  9. If the ZooKeeper service starts successfully, stop the service by using:

    sudo /opt/zookeeper-<release_version>/zookeeper-wrapper/bin/zookeeper stop

(Optional) Configuring the Maximum Heap Size for ZooKeeper

Configure the max heap size ( Xmx ) for ZooKeeper by uncommenting the following line in /opt/zookeeper-<release_version/zookeeper-wrapper/conf/wrapper.conf :

#wrapper.java.maxmemory=64

For example to set a 2 GB heap size, use:

wrapper.java.maxmemory=2048

Otherwise, by default as per https://zookeeper.apache.org/doc/r3.3.4/zookeeperAdmin.html, it uses a maximum of 256 MB (if the physical RAM exceeds 1 GB). See the memory recommendation in https://zookeeper.apache.org/doc/r3.3.4/zookeeperAdmin.html .

Configuring ZooKeeper Service Auto-Restart on Linux

Perform the following procedures to auto-restart the ZooKeeper process upon server reboot or when the process exits abnormally.

Installing daemontools

You can untar daemontools-0.76.tar.gz into any directory you want, but do not rename that directory after you have done your first package/install command. If you were to rename it after that first package/install command, daemontools would silently fail. The build actually queries for the current directory and writes it into various compile files.

  • Log in as root.

  • Enter the following commands:

    mkdir -p /package
    chmod 1755 /package
    cd /package
  • Download daemontools by entering:

    wget \https://cr.yp.to/daemontools/daemontools-0.76.tar.gz

    or

    curl -o daemontools-0.76.tar.gz \https://cr.yp.to/daemontools/daemontools-0.76.tar.gz
  • Install daemontools by entering:

    tar -xpf daemontools-0.76.tar.gz
    rm -f daemontools-0.76.tar.gz
    yum -y install gcc
    cd admin/daemontools-0.76
  • In /package/admin/daemontools-0.76/src/conf-cc, change

    gcc -O2 -Wimplicit -Wunused -Wcomment -Wchar-subscripts -Wuninitialized -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings

    to

    gcc -O2 -Wimplicit -Wunused -Wcomment -Wchar-subscripts -Wuninitialized -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings -include errno.h
  • Run:

    package/install

    Here is the tail end of the output:

    The command tools are in /package/admin/daemontools-0.76/command.

Configuring Auto-Restart Using daemontools

This section assumes that you installed ZooKeeper into /opt/zookeeper-3.4.10.

  • Enter the following commands:

    mkdir -p /opt/zookeeper-3.4.10/service/zookeeper
    cd /opt/zookeeper-3.4.10/service/zookeeper
  • Create a file named /opt/zookeeper-3.4.10/service/zookeeper/run with the following contents:

    #!/bin/sh echo Starting hello exec /opt/zookeeper-3.4.10/zookeeper-wrapper/bin/zookeeper start
  • Grant execute permissions by entering:

    sudo chmod u+x run
  • Create a soft link under the /service folder that will be monitored by daemontools by entering:

    ln -s /opt/zookeeper-3.4.10/service/zookeeper /service/zookeeper

    Note that this command tries to “install” the zookeeper service but will fail, because as per link:https://cr.yp.to/daemontools/start.html\https://cr.yp.to/daemontools/start.html , it adds the following entry to /etc/inittab :

     SV:12345:respawn:/command/svscanboot
  • Resolve the zookeeper service failure issue by opening /etc/inittab and commenting out the line that you added above.

    In other words, change

    SV:12345:respawn:/command/svscanboot

    to

    #SV:12345:respawn:/command/svscanboot

Configuring Auto-Restart Using daemontools (RHEL 7 and CentOS 7)

RHEL 7 and Centos 7 use systemd for managing services.

  • Create a file named /etc/systemd/system/daemontools.service with the following startup code in it:

    [Unit]
    Description=daemontools Start supervise
    After=getty.target
    [Service]
    Type=simple
    User=root
    Group=root
    Restart=always
    ExecStart=/command/svscanboot /dev/ttyS0
    TimeoutSec=0
    
    [Install]
    WantedBy=multi-user.target
  • Start the daemontools service by entering:

     systemctl start daemontools.service
  • Test that the daemontools service is running by entering:

     systemctl status daemontools.service

    From the output, you can see that the ZooKeeper service has started.

  • Enable the daemontools service to start at boot time by entering:

    systemctl enable daemontools.service

For more information about using systemd to configure auto-restart, see link:https://www.productionmonkeys.net/guides/qmail-server/daemontools\https://www.productionmonkeys.net/guides/qmail-server/daemontools .

Process That are Started by daemontools

Daemontools starts two processes:

  • The zookeeper-wrapper process

  • The child zookeeper Java process started by zookeeper-wrapper

If you kill the Java process, the zookeeper-wrapper process restarts it. If you kill the zookeeper-wrapper process, then daemontools.service will start it. In this way, the zookeeper process will auto-start on server reboot and also when the process exits abnormally.

Useful Folders

FolderPurpose

/package/admin/daemontools-0.76/

Folder where daemontools-0.76 is installed

/opt/zookeeper-3.4.10/service/zookeeper

Folder with the service run script

/opt/zookeeper-3.4.10/zookeeper-wrapper

ZooKeeper service wrapper

Troubleshooting Resources

  • https://www.troubleshooters.com/linux/djbdns/daemontools_intro.htm#elementary_troubleshooting

  • https://isotope11.com/blog/manage-your-services-with-daemontools

For More Information

Steps to install daemontools:

  • https://isotope11.com/blog/manage-your-services-with-daemontools

  • https://www.troubleshooters.com/linux/djbdns/daemontools_intro.htm

Other ways to monitor ZooKeeper:

  • As per https://blog.serverdensity.com/how-to-monitor-zookeeper/:

    $ ./zktop.py --servers "localhost:2181,localhost:2182,localhost:2183"
  • https://docs.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_ig_zookeeper_installation.html

  • https://supervisord.org/ (shares some of the same goals of programs such as launchd, daemontools, and runit)

Running ZooKeeper as a Service on Windows

Install ZooKeeper on machines other than those running CloudBees Flow. ZooKeeper must not be run on the same machines as those running the CloudBees Flow servers.

To run ZooKeeper as a service, follow these steps:

  1. For each machine on which you want to install ZooKeeper, navigate to the zookeeper-wrapper.zip file in the CloudBees Flow <install dir>\utils directory.

  2. Copy or move the zookeeper-wrapper.zip file to each of the machines that you want to use for ZooKeeper.

  3. Extract zookeeper-wrapper.zip to a directory of your choice.

    For example, ` C:\Users\Administrator\zooservice`

  4. Using a text editor, open the wrapper.conf file located in the extracted zookeeper-wrapper\conf directory.

    For example, C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf

  5. Add the path to the ZooKeeper home directory.

    For example:

    # Path to unpacked zookeeper

    set.default.ZOOKEEPER_HOME=C:\Users\Administrator\zooservice\zookeeper-3.4.6

    Now you are ready to install and start ZooKeeper as a service.

  6. Navigate to and click InstallZooKeeper-NT.bat to install ZooKeeper as a service.

    The file is in the zookeeper-wrapper\bin directory.

    For example, ` C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf`.

  7. Navigate to and click StartZooKeeper-NT.bat to start ZooKeeper as a service.

    The file is located in the zookeeper-wrapper\bin directory.

    For example, ` C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf`.

    If you choose to use the command- line interface or a script to start the service, enter ZooKeeperCommand.bat start.

Ensuring that ZooKeeper Can Locate Java

Because ZooKeeper is a Java application, ensure Java is installed and ZooKeeper can locate it. The default value for the JAVA_HOME setting (in the zookeeper-wrapper/conf/wrapper.conf file) is

set.default.JAVA_HOME=/opt/electriccloud/electriccommander/jre

If CloudBees Flow is not installed or is not installed in its default directory, set JAVA_HOME in wrapper.conf to the location for Java. For example:

set.default.JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

Verifying that ZooKeeper is in Standalone Mode

The following example shows how to verify that the Zookeeper service is running in standalone mode:

/opt/zookeeper-3.4.5/bin$sudo ./zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Mode: standalone
You can find the Zookeeper bin directory by running ps -ef grep "zoo.cfg". This command will display the location of the zoo.cfg file. For example, zookeeper-<release_version>/conf/zoo.cfg.

If you do not see the status above but see the error below, then ZooKeeper might be configured in replication mode and therefore cannot connect to the other nodes in its ensemble:

JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

The status when ZooKeeper is in replication mode will look something like:

Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
Clients:
10.168.33.13.35821[0](queued=0,recved=1, sent=0)
10.68.33.13.35748[1](queued=0,recved=2189,sent=2189)

Latency min/avg/max: 0/0/86
Received: 2198
Sent: 2197
Connections: 2
Outstanding: 0
Zxid: Ox27758
Mode: standalone
Node count: 29

In this case, you must configure Zookeeper in standalone mode and then restart the ZooKeeper service as in the following example:

/opt/zookeeper-3.4.5/bin$sudo./zkServer.sh stop
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Stopping zookeeper ..../zkServer.sh: line 143: kill: (1776) - No such process
STOPPED
/opt/zookeeper-3.4.5/bin$sudo./zkServer.sh start
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

Verifying that ZooKeeper is Running

To check that the ZooKeeper software is running, follow these steps:

  1. Log in to each ZooKeeper machine and enter:

    echo ruok | nc 127.0.0.1 2181

  2. Confirm that you get the following response from each ZooKeeper instance by entering:

    imok

If no response appears or a broken pipe error appears, then ZooKeeper is not running.

  1. Obtain more information about the status of Zookeeper by logging into each ZooKeeper machine and entering:

    echo status | nc 127.0.0.1 2181

Exhibitor Software

After installing ZooKeeper consider installing the optional Exhibitor software on every machine with an instance of Zookeeper. The Exhibitor software provides a web interface that allows you to monitor the status of ZooKeeper. It also keeps the configurations of all the ZooKeeper nodes in sync when any of them are changed, and provides tools to rotate and prune the ZooKeeper logs, to prevent them from growing indefinitely. For more information, see the Exhibitor documentation at https://github.com/Netflix/exhibitor.

If you choose to install Exhibitor, you must configure your CloudBees Flow server nodes so they know how to contact Exhibitor. If you have already set up your CloudBees Flow server cluster and ZooKeeper servers and are later adding Exhibitor to it, this can be done using the ecconfigure tool, which is normally found at /opt/electriccloud/electriccommander/bin/ecconfigure on Linux or C:\Program Files\Electric Cloud\ElectricCommander\bin\ecconfigure.exe on Windows. Bring down all nodes in your cluster, and run ecconfigure on each CloudBees Flow node in the cluster with the option (as the user that CloudBees Flow runs as, or with administrative privileges):

ecconfigure --serverExhibitorConnection <Exhibitor_servers>

where <Exhibitor_servers> is a comma-separated (no spaces) list of the IP_address_or_FQDN:port_number of your three or five (or for a test system, possibly just one) Exhibitor servers (the port number of Exhibitor is normally 8080). For example, 10.0.2.1:8080,10.0.2.2:8080,10.0.2.3:8080 for a three-ZooKeeper/Exhibitor cluster.

It is not necessary to use the command with a single exhibitor and a single ZooKeeper server.