Install ZooKeeper

6 minute readScalability

Use Apache ZooKeeper v3.8.3 or later to maintain and synchronize group services in a clustered CloudBees CD/RO configuration. For more information, refer to the Apache ZooKeeper website.

For your convenience, ZooKeeper v3.8.3 is bundled in your CloudBees CD/RO installation here: <install dir>/utils.

To install ZooKeeper:

  1. Use either the bundled ZooKeeper or download ZooKeeper from the ZooKeeper website.

  2. Extract and install the files into an appropriate location. For example, /opt/zookeeper-3.8.3

    You must install ZooKeeper on an odd number of machines. The number of machines will determine if you install ZooKeeper in standalone mode (for one machine) or in replicated mode (for three or five machines).
  3. Create a zoo.cfg configuration file for each machine with an instance of ZooKeeper. For example, zookeeper-3.8.3/conf/zoo.cfg.

    • For standalone mode on a single ZooKeeper machine, the file has these values:

      tickTime=2000
      dataDir=/var/lib/zookeeper
      clientPort=2181
    • For replicated mode across multiple ZooKeeper machines, the zoo.cfg file on each server has these values:

      tickTime=2000
      dataDir=/var/lib/zookeeper
      clientPort=2181
      initLimit=5
      syncLimit=2
      server.1=<ZooKeeper_hostname_1>:2888:3888
      server.2=<ZooKeeper_hostname_2>:2888:3888
      server.3=<ZooKeeper_hostname_3>:2888:3888
      server.4=<ZooKeeper_hostname_4>:2888:3888
      server.5=<ZooKeeper_hostname_5>:2888:3888

      Where < ZooKeeper_hostname_1 > through < ZooKeeper_hostname_5 > are the hostnames of the servers for the ZooKeeper service.

      The file has only three server value lines for a three-ZooKeeper configuration.
  4. Create /var/lib/zookeeper/myid files on each ZooKeeper server, with each containing a single ASCII digit: 1 for the first server, 2 for the second server, and so on, corresponding to their server. <digit> values in the zoo.cfg files

  5. Create /var/lib/zookeeper/myid files on each ZooKeeper server, with each containing a single ASCII digit as follows:

    • 1 for the first server

    • 2 for the second server

    • Up to 5, corresponding to the appropriate server. <digit> value in the zoo.cfg file.

  6. Start ZooKeeper by entering the following command:

Missing command

Running ZooKeeper as a service on Linux

Install ZooKeeper on machines other than those running CloudBees CD/RO. ZooKeeper must not be run on the same machines as those running the CloudBees CD/RO servers.

To run ZooKeeper as a service (running as the root user), follow these steps:

  1. For each machine on which you want to install ZooKeeper, navigate to the zookeeper-wrapper.zip file in the CloudBees CD/RO <install dir>/utils directory.

  2. Copy or move the zookeeper-wrapper.zip file to each of the machines that you want to use for ZooKeeper.

  3. Extract zookeeper-wrapper.zip to a directory of your choice.

    For example, /opt/zookeeper-3.8.3.

  4. Using a text editor, open the wrapper.conf file in the extracted zookeeper-wrapper/conf directory.

    For example, /opt/zookeeper-3.8.3/zookeeper-wrapper/conf.

  5. Add the path to the ZooKeeper home directory.

    # Path to unpacked zookeeper
    
    set.default.ZOOKEEPER_HOME=/opt/zookeeper-<ZK-VERSION>

    For example:

    # Path to unpacked zookeeper
    
    set.default.ZOOKEEPER_HOME=/opt/zookeeper-3.8.3
  6. Add the path to the JRE. For example:

    set.default.JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
    ZooKeeper has specific OS limitations and JDK version requirements. For more information, refer to Apache ZooKeeper.
  7. Edit /opt/zookeeper-3.8.3/zookeeper-wrapper/bin/zookeeper to change:

    WRAPPER_ROOT="../"

    to

    WRAPPER_ROOT="/opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper"

    For example:

    WRAPPER_ROOT="/opt/zookeeper-3.8.3/zookeeper-wrapper"
  8. Verify that you can start the ZooKeeper service by running:

    sudo /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/bin/zookeeper start

    For example:

    sudo /opt/zookeeper-3.8.3/zookeeper-wrapper/bin/zookeeper start
  9. If the ZooKeeper service starts successfully, you can stop the service using:

    sudo /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/bin/zookeeper stop

    For example:

    sudo /opt/zookeeper-3.8.3/zookeeper-wrapper/bin/zookeeper stop

(Optional) Configuring the maximum heap size for ZooKeeper

Configure the max heap size ( Xmx ) for ZooKeeper by uncommenting the following line in /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/conf/wrapper.conf :

#wrapper.java.maxmemory=64

For example to set a 2 GB heap size, use:

wrapper.java.maxmemory=2048

Otherwise, by default as per ZooKeeper Administrator’s Guide, it uses a maximum of 256 MB (if the physical RAM exceeds 1 GB). Refer to the memory recommendation in ZooKeeper Administrator’s Guide.

Configuring the ZooKeeper service auto-restart on Linux

Perform the following procedures to auto-restart the ZooKeeper process upon server reboot or when the process exits abnormally.

Installing daemontools

You can untar daemontools-0.76.tar.gz into any directory you want, but do not rename that directory after you have done your first package/install command. If you were to rename it after that first package/install command, daemontools would silently fail. The build actually queries for the current directory and writes it into various compile files.

  1. Log in as root.

  2. Enter the following commands:

    mkdir -p /package
    chmod 1755 /package
    cd /package
  3. Download daemontools by entering:

    wget https://cr.yp.to/daemontools/daemontools-0.76.tar.gz

    or

    curl -o daemontools-0.76.tar.gz https://cr.yp.to/daemontools/daemontools-0.76.tar.gz
  4. Install daemontools by entering:

    tar -xpf daemontools-0.76.tar.gz
    rm -f daemontools-0.76.tar.gz
    yum -y install gcc
    cd admin/daemontools-0.76
  5. In /package/admin/daemontools-0.76/src/conf-cc, change:

    gcc -O2 -Wimplicit -Wunused -Wcomment -Wchar-subscripts -Wuninitialized -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings

    to:

    gcc -O2 -Wimplicit -Wunused -Wcomment -Wchar-subscripts -Wuninitialized -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings -include errno.h
  6. Run:

    package/install

    The end of the output should look simuliar to:

    ...
    Coping commands into ./command...
    Creating symlink daemontools -> daemontools-0.76...
    Making command links in /command...
    Making compatibility links in /usr/local/bin...
    Creating /service...
    Adding svscanboot to inittab...
    inti should start svscan now.

    The command tools are in /package/admin/daemontools-0.76/command.

Configuring auto-restart using daemontools

You can use daemontools to monitor and start ZooKeeper.

  1. Run the following commands to create a ZooKeeper service directory and navigate to it:

    mkdir -p /opt/zookeeper-<ZK-VERSION>/service/zookeeper
    cd /opt/zookeeper-<ZK-VERSION>/service/zookeeper

    For example:

    mkdir -p /opt/zookeeper-3.8.3/service/zookeeper
    cd /opt/zookeeper-3.8.3/service/zookeeper
  2. Create a file named /opt/zookeeper-<ZK-VERSION>/service/zookeeper/run with the following contents:

    #!/bin/sh
    echo Starting hello
    exec /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/bin/zookeeper start

    For example:

    #!/bin/sh
    echo Starting hello
    exec /opt/zookeeper-3.8.3/zookeeper-wrapper/bin/zookeeper start
  3. Grant execute permissions by entering:

    sudo chmod u+x run
  4. Create a soft link under the /service folder that will be monitored by daemontools by entering:

    ln -s /opt/zookeeper-<ZK-VERSION>/service/zookeeper /service/zookeeper

    For example:

    ln -s /opt/zookeeper-3.8.3/service/zookeeper /service/zookeeper
    This command attempts to “install” the zookeeper service but fails, because it adds the following entry to /etc/inittab :
    SV:12345:respawn:/command/svscanboot
  5. To resolve this issue, open /etc/inittab:

    Search for:

    SV:12345:respawn:/command/svscanboot

    Comment it out:

    #SV:12345:respawn:/command/svscanboot

Configuring auto-restart using daemontools (RHEL 7 and CentOS 7)

RHEL 7 and Centos 7 use systemd for managing services.

  1. Create a file named /etc/systemd/system/daemontools.service with the following startup code in it:

    [Unit]
    Description=daemontools Start supervise
    After=getty.target
    [Service]
    Type=simple
    User=root
    Group=root
    Restart=always
    ExecStart=/command/svscanboot /dev/ttyS0
    TimeoutSec=0
    
    [Install]
    WantedBy=multi-user.target
  2. Start the daemontools service by entering:

    systemctl start daemontools.service
  3. Test that the daemontools service is running by entering:

    systemctl status daemontools.service

    From the output, you can confirm that the ZooKeeper service has started.

  4. Enable the daemontools service to start at boot time by entering:

    systemctl enable daemontools.service

Processes started by daemontools

Daemontools starts two processes:

  • The zookeeper-wrapper process

  • The child zookeeper Java process started by zookeeper-wrapper

If you terminate the Java process, the zookeeper-wrapper process restarts it. If you terminate the zookeeper-wrapper process, the daemontools.service starts it. In this way, the zookeeper process will auto-start on server reboot and also when the process exits abnormally.

Useful folders

Folder Purpose

/package/admin/daemontools-0.76/

Folder where daemontools-0.76 is installed

/opt/zookeeper-<ZK-VERSION>/service/zookeeper

Folder with the service run script

/opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper

ZooKeeper service wrapper

Running ZooKeeper as a service on Windows

Install ZooKeeper on machines other than those running CloudBees CD/RO. ZooKeeper must not be run on the same machines as those running the CloudBees CD/RO servers.

To run ZooKeeper as a service, follow these steps:

  1. For each machine on which you want to install ZooKeeper, navigate to the zookeeper-wrapper.zip file in the CloudBees CD/RO <install dir>\utils directory.

  2. Copy or move the zookeeper-wrapper.zip file to each of the machines that you want to use for ZooKeeper.

  3. Extract zookeeper-wrapper.zip to a directory.

    For example, C:\Users\Administrator\zooservice

  4. Using a text editor, open the wrapper.conf file located in the extracted zookeeper-wrapper\conf directory.

    For example:

    C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf
  5. Add the path to the ZooKeeper home directory.

    # Path to unpacked zookeeper
    
    set.default.ZOOKEEPER_HOME=C:\Users\Administrator\zooservice\zookeeper-<ZK-VERSION>

    For example:

    # Path to unpacked zookeeper
    
    set.default.ZOOKEEPER_HOME=C:\Users\Administrator\zooservice\zookeeper-3.8.3

    Now you are ready to install and start ZooKeeper as a service.

  6. Navigate to and click InstallZooKeeper-NT.bat to install ZooKeeper as a service.

    The file is in the zookeeper-wrapper\bin directory.

    For example:

    C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf
  7. Navigate to and click StartZooKeeper-NT.bat to start ZooKeeper as a service.

    The file is located in the zookeeper-wrapper\bin directory.

    For example:

    C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf

    If you choose to use the command-line interface or a script to start the service, enter:

    ZooKeeperCommand.bat start

Ensuring that ZooKeeper can locate Java

Because ZooKeeper is a Java application, ensure Java is installed and ZooKeeper can locate it. The default value for the JAVA_HOME setting (in the zookeeper-wrapper/conf/wrapper.conf file) is:

set.default.JAVA_HOME=/opt/cloudbees/sda/jre

If CloudBees CD/RO is not installed or is not installed in its default directory, set JAVA_HOME in wrapper.conf to the location for Java. For example:

set.default.JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

Verifying that ZooKeeper is in standalone mode

The following example shows how to verify that the ZooKeeper service is running in standalone mode:

You can find the ZooKeeper bin directory by running ps -ef | grep "zoo.cfg". This command will display the location of the zoo.cfg file. For example, zookeeper-3.8.3/conf/zoo.cfg.

Get the ZooKeeper status:

/opt/zookeeper-<ZK-VERSION>/bin$sudo ./zkServer.sh status

For example:

/opt/zookeeper-3.8.3/bin$sudo ./zkServer.sh status

The response should be similar to:

JMX enabled by default
Using config: /opt/zookeeper-3.8.3/bin/../conf/zoo.cfg
Mode: standalone

If you receive the following error:

JMX enabled by default
Using config: /opt/zookeeper-3.8.3/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

ZooKeeper might be configured in replication mode and therefore cannot connect to the other nodes in its ensemble.

The status when ZooKeeper is in replication mode will look similar to the following:

ZooKeeper version: 3.8.3-1392090, built on 03/30/2023 17:52 GMT
Clients:
10.168.33.13.35821[0](queued=0,recved=1, sent=0)
10.68.33.13.35748[1](queued=0,recved=2189,sent=2189)

Latency min/avg/max: 0/0/86
Received: 2198
Sent: 2197
Connections: 2
Outstanding: 0
Zxid: Ox27758
Mode: standalone
Node count: 29

In this case, you must configure ZooKeeper in standalone mode and then restart the ZooKeeper service:

  1. To stop ZooKeeper, run:

    /opt/zookeeper-<ZK-VERSION>/bin$sudo./zkServer.sh stop

    For example:

    /opt/zookeeper-3.8.3/bin$sudo./zkServer.sh stop
    • The response should be similar to:

      JMX enabled by default
      Using config: /opt/zookeeper-3.8.3/bin/../conf/zoo.cfg
      Stopping zookeeper ..../zkServer.sh: line 143: kill: (1776) - No such process
      STOPPED
  2. To restart ZooKeeper, run:

    /opt/zookeeper-<ZK-VERSION>/bin$sudo./zkServer.sh start

    For example:

    /opt/zookeeper-3.8.3/bin$sudo./zkServer.sh start
    • The response should be similar to:

      JMX enabled by default
      Using config: /opt/zookeeper-3.8.3/bin/../conf/zoo.cfg
      Starting zookeeper ... STARTED

Verifying that ZooKeeper is running

For testing or troubleshooting purposes, it may be necessary to verify the ZooKeeper server is running. To verify ZooKeeper is running:

  1. Configure ZooKeeper to recognize four-character test commands:

    1. Open the <zookeeper>/conf/zoo.zfg file.

    2. Modify the configuration file one of the following ways:

      • To allow test commands only enter: 4lw.commands.whitelist=stat, ruok, conf, isro

      • To authorize all four-character commands enter: 4lw.commands.whitelist=*

  2. Log in to each ZooKeeper machine and run:

    echo ruok | nc 127.0.0.1 2181
  3. Confirm that you get a response from each ZooKeeper instance by running:

    imok
    • If no response appears or a broken pipe error appears, then ZooKeeper is not running.

  4. Obtain more information about the status of ZooKeeper by logging into each ZooKeeper machine and running:

    echo status | nc 127.0.0.1 2181