Install ZooKeeper

Use Apache ZooKeeper v3.8.4 or later to maintain and synchronize group services in a clustered CloudBees CD/RO configuration. For more information, refer to the Apache ZooKeeper website.

For your convenience, ZooKeeper v3.8.4 is bundled in your CloudBees CD/RO installation here: <install dir>/utils.

To install ZooKeeper:

Use either the bundled ZooKeeper or download ZooKeeper from the ZooKeeper website.

Extract and install the files into an appropriate location. For example, /opt/zookeeper-3.8.4

You must install ZooKeeper on an odd number of machines. The number of machines will determine if you install ZooKeeper in standalone mode (for one machine) or in replicated mode (for three or five machines).

Create a zoo.cfg configuration file for each machine with an instance of ZooKeeper. For example, zookeeper-3.8.4/conf/zoo.cfg.
- For standalone mode on a single ZooKeeper machine, the file has these values:
  tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181
  ▼
- For replicated mode across multiple ZooKeeper machines, the zoo.cfg file on each server has these values:
  tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=<ZooKeeper_hostname_1>:2888:3888 server.2=<ZooKeeper_hostname_2>:2888:3888 server.3=<ZooKeeper_hostname_3>:2888:3888 server.4=<ZooKeeper_hostname_4>:2888:3888 server.5=<ZooKeeper_hostname_5>:2888:3888
  ▼
  Where < ZooKeeper_hostname_1 > through < ZooKeeper_hostname_5 > are the hostnames of the servers for the ZooKeeper service.
  
  The file has only three server value lines for a three-ZooKeeper configuration.
Create /var/lib/zookeeper/myid files on each ZooKeeper server, with each containing a single ASCII digit: 1 for the first server, 2 for the second server, and so on, corresponding to their server. <digit> values in the zoo.cfg files
Create /var/lib/zookeeper/myid files on each ZooKeeper server, with each containing a single ASCII digit as follows:
- 1 for the first server
- 2 for the second server
- Up to 5, corresponding to the appropriate server. <digit> value in the zoo.cfg file.
Start ZooKeeper by entering the following command:

Missing command

Running ZooKeeper as a service on Linux

Install ZooKeeper on machines other than those running CloudBees CD/RO. ZooKeeper must not be run on the same machines as those running the CloudBees CD/RO servers.

To run ZooKeeper as a service (running as the root user), follow these steps:

For each machine on which you want to install ZooKeeper, navigate to the zookeeper-wrapper.zip file in the CloudBees CD/RO <install dir>/utils directory.
Copy or move the zookeeper-wrapper.zip file to each of the machines that you want to use for ZooKeeper.
Extract zookeeper-wrapper.zip to a directory of your choice.

For example, /opt/zookeeper-3.8.4.
Using a text editor, open the wrapper.conf file in the extracted zookeeper-wrapper/conf directory.

For example, /opt/zookeeper-3.8.4/zookeeper-wrapper/conf.

Add the path to the ZooKeeper home directory.

# Path to unpacked zookeeper

set.default.ZOOKEEPER_HOME=/opt/zookeeper-<ZK-VERSION>▼

For example:

# Path to unpacked zookeeper

set.default.ZOOKEEPER_HOME=/opt/zookeeper-3.8.4▼

Add the path to the JRE. For example:
```
set.default.JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre▼
```
ZooKeeper has specific OS limitations and JDK version requirements. For more information, refer to Apache ZooKeeper.

Edit /opt/zookeeper-3.8.4/zookeeper-wrapper/bin/zookeeper to change:

WRAPPER_ROOT="../"▼

WRAPPER_ROOT="/opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper"▼

For example:

WRAPPER_ROOT="/opt/zookeeper-3.8.4/zookeeper-wrapper"▼

Verify that you can start the ZooKeeper service by running:

sudo /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/bin/zookeeper start▼

For example:

sudo /opt/zookeeper-3.8.4/zookeeper-wrapper/bin/zookeeper start▼

If the ZooKeeper service starts successfully, you can stop the service using:

sudo /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/bin/zookeeper stop▼

For example:

sudo /opt/zookeeper-3.8.4/zookeeper-wrapper/bin/zookeeper stop▼

(Optional) Configuring the maximum heap size for ZooKeeper

Configure the max heap size ( Xmx ) for ZooKeeper by uncommenting the following line in /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/conf/wrapper.conf :

#wrapper.java.maxmemory=64▼

For example to set a 2 GB heap size, use:

wrapper.java.maxmemory=2048▼

Otherwise, by default as per ZooKeeper Administrator’s Guide, it uses a maximum of 256 MB (if the physical RAM exceeds 1 GB). Refer to the memory recommendation in ZooKeeper Administrator’s Guide.

Configuring the ZooKeeper service auto-restart on Linux

Perform the following procedures to auto-restart the ZooKeeper process upon server reboot or when the process exits abnormally.

Installing daemontools

You can untar daemontools-0.76.tar.gz into any directory you want, but do not rename that directory after you have done your first package/install command. If you were to rename it after that first package/install command, daemontools would silently fail. The build actually queries for the current directory and writes it into various compile files.

Enter the following commands:

mkdir -p /package
chmod 1755 /package
cd /package▼

Download daemontools by entering:

wget https://cr.yp.to/daemontools/daemontools-0.76.tar.gz▼

curl -o daemontools-0.76.tar.gz https://cr.yp.to/daemontools/daemontools-0.76.tar.gz▼

Install daemontools by entering:

tar -xpf daemontools-0.76.tar.gz
rm -f daemontools-0.76.tar.gz
yum -y install gcc
cd admin/daemontools-0.76▼

In /package/admin/daemontools-0.76/src/conf-cc, change:

gcc -O2 -Wimplicit -Wunused -Wcomment -Wchar-subscripts -Wuninitialized -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings▼

to:

gcc -O2 -Wimplicit -Wunused -Wcomment -Wchar-subscripts -Wuninitialized -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings -include errno.h▼

Run:

package/install▼

The end of the output should look simuliar to:

...
Coping commands into ./command...
Creating symlink daemontools -> daemontools-0.76...
Making command links in /command...
Making compatibility links in /usr/local/bin...
Creating /service...
Adding svscanboot to inittab...
inti should start svscan now.▼

The command tools are in /package/admin/daemontools-0.76/command.

Configuring auto-restart using daemontools

You can use daemontools to monitor and start ZooKeeper.

Run the following commands to create a ZooKeeper service directory and navigate to it:

mkdir -p /opt/zookeeper-<ZK-VERSION>/service/zookeeper
cd /opt/zookeeper-<ZK-VERSION>/service/zookeeper▼

For example:

mkdir -p /opt/zookeeper-3.8.4/service/zookeeper
cd /opt/zookeeper-3.8.4/service/zookeeper▼

Create a file named /opt/zookeeper-<ZK-VERSION>/service/zookeeper/run with the following contents:

#!/bin/sh
echo Starting hello
exec /opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper/bin/zookeeper start▼

For example:

#!/bin/sh
echo Starting hello
exec /opt/zookeeper-3.8.4/zookeeper-wrapper/bin/zookeeper start▼

Grant execute permissions by entering:
```
sudo chmod u+x run▼
```
Create a soft link under the /service folder that will be monitored by daemontools by entering:
```
ln -s /opt/zookeeper-<ZK-VERSION>/service/zookeeper /service/zookeeper▼
```
For example:
```
ln -s /opt/zookeeper-3.8.4/service/zookeeper /service/zookeeper▼
```
This command attempts to “install” the zookeeper service but fails, because it adds the following entry to /etc/inittab :
```
SV:12345:respawn:/command/svscanboot▼
```

To resolve this issue, open /etc/inittab:

Search for:

SV:12345:respawn:/command/svscanboot▼

Comment it out:

#SV:12345:respawn:/command/svscanboot▼

Configuring auto-restart using daemontools (RHEL 7 and CentOS 7)

RHEL 7 and Centos 7 use systemd for managing services.

Create a file named /etc/systemd/system/daemontools.service with the following startup code in it:

[Unit]
Description=daemontools Start supervise
After=getty.target
[Service]
Type=simple
User=root
Group=root
Restart=always
ExecStart=/command/svscanboot /dev/ttyS0
TimeoutSec=0

[Install]
WantedBy=multi-user.target▼

Start the daemontools service by entering:
```
systemctl start daemontools.service▼
```
Test that the daemontools service is running by entering:
```
systemctl status daemontools.service▼
```
From the output, you can confirm that the ZooKeeper service has started.
Enable the daemontools service to start at boot time by entering:
```
systemctl enable daemontools.service▼
```

Processes started by daemontools

Daemontools starts two processes:

The zookeeper-wrapper process
The child zookeeper Java process started by zookeeper-wrapper

If you terminate the Java process, the zookeeper-wrapper process restarts it. If you terminate the zookeeper-wrapper process, the daemontools.service starts it. In this way, the zookeeper process will auto-start on server reboot and also when the process exits abnormally.

Useful folders

Folder	Purpose
/package/admin/daemontools-0.76/	Folder where daemontools-0.76 is installed
/opt/zookeeper-<ZK-VERSION>/service/zookeeper	Folder with the service run script
/opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper	ZooKeeper service wrapper

Folder

Purpose

/package/admin/daemontools-0.76/

Folder where daemontools-0.76 is installed

/opt/zookeeper-<ZK-VERSION>/service/zookeeper

Folder with the service run script

/opt/zookeeper-<ZK-VERSION>/zookeeper-wrapper

ZooKeeper service wrapper

Running ZooKeeper as a service on Windows

Install ZooKeeper on machines other than those running CloudBees CD/RO. ZooKeeper must not be run on the same machines as those running the CloudBees CD/RO servers.

To run ZooKeeper as a service, follow these steps:

For each machine on which you want to install ZooKeeper, navigate to the zookeeper-wrapper.zip file in the CloudBees CD/RO <install dir>\utils directory.
Copy or move the zookeeper-wrapper.zip file to each of the machines that you want to use for ZooKeeper.
Extract zookeeper-wrapper.zip to a directory.

For example, C:\Users\Administrator\zooservice
Using a text editor, open the wrapper.conf file located in the extracted zookeeper-wrapper\conf directory.

For example:
```
C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf▼
```

Add the path to the ZooKeeper home directory.

# Path to unpacked zookeeper

set.default.ZOOKEEPER_HOME=C:\Users\Administrator\zooservice\zookeeper-<ZK-VERSION>▼

For example:

# Path to unpacked zookeeper

set.default.ZOOKEEPER_HOME=C:\Users\Administrator\zooservice\zookeeper-3.8.4▼

Now you are ready to install and start ZooKeeper as a service.

Navigate to and click InstallZooKeeper-NT.bat to install ZooKeeper as a service.

The file is in the zookeeper-wrapper\bin directory.

For example:
```
C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf▼
```
Navigate to and click StartZooKeeper-NT.bat to start ZooKeeper as a service.

The file is located in the zookeeper-wrapper\bin directory.

For example:
```
C:\Users\Administrator\zooservice\zookeeper-wrapper\zookeeper-wrapper\conf▼
```
If you choose to use the command-line interface or a script to start the service, enter:
```
ZooKeeperCommand.bat start▼
```

Ensuring that ZooKeeper can locate Java

Because ZooKeeper is a Java application, ensure Java is installed and ZooKeeper can locate it. The default value for the JAVA_HOME setting (in the zookeeper-wrapper/conf/wrapper.conf file) is:

set.default.JAVA_HOME=/opt/cloudbees/sda/jre▼

If CloudBees CD/RO is not installed or is not installed in its default directory, set JAVA_HOME in wrapper.conf to the location for Java. For example:

set.default.JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64▼

Verifying that ZooKeeper is in standalone mode

The following example shows how to verify that the ZooKeeper service is running in standalone mode:

You can find the ZooKeeper bin directory by running ps -ef | grep "zoo.cfg". This command will display the location of the zoo.cfg file. For example, zookeeper-3.8.4/conf/zoo.cfg.

Get the ZooKeeper status:

/opt/zookeeper-<ZK-VERSION>/bin$sudo ./zkServer.sh status▼

For example:

/opt/zookeeper-3.8.4/bin$sudo ./zkServer.sh status▼

The response should be similar to:

JMX enabled by default
Using config: /opt/zookeeper-3.8.4/bin/../conf/zoo.cfg
Mode: standalone▼

If you receive the following error:

JMX enabled by default
Using config: /opt/zookeeper-3.8.4/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.▼

ZooKeeper might be configured in replication mode and therefore cannot connect to the other nodes in its ensemble.

The status when ZooKeeper is in replication mode will look similar to the following:

ZooKeeper version: 3.8.4-1392090, built on 03/30/2023 17:52 GMT
Clients:
10.168.33.13.35821[0](queued=0,recved=1, sent=0)
10.68.33.13.35748[1](queued=0,recved=2189,sent=2189)

Latency min/avg/max: 0/0/86
Received: 2198
Sent: 2197
Connections: 2
Outstanding: 0
Zxid: Ox27758
Mode: standalone
Node count: 29▼

In this case, you must configure ZooKeeper in standalone mode and then restart the ZooKeeper service:

To stop ZooKeeper, run:

/opt/zookeeper-<ZK-VERSION>/bin$sudo./zkServer.sh stop▼

For example:

/opt/zookeeper-3.8.4/bin$sudo./zkServer.sh stop▼

The response should be similar to:

JMX enabled by default
Using config: /opt/zookeeper-3.8.4/bin/../conf/zoo.cfg
Stopping zookeeper ..../zkServer.sh: line 143: kill: (1776) - No such process
STOPPED▼

To restart ZooKeeper, run:

/opt/zookeeper-<ZK-VERSION>/bin$sudo./zkServer.sh start▼

For example:

/opt/zookeeper-3.8.4/bin$sudo./zkServer.sh start▼

The response should be similar to:

JMX enabled by default
Using config: /opt/zookeeper-3.8.4/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED▼

Verifying that ZooKeeper is running

For testing or troubleshooting purposes, it may be necessary to verify the ZooKeeper server is running. To verify ZooKeeper is running:

Configure ZooKeeper to recognize four-character test commands:
1. Open the <zookeeper>/conf/zoo.cfg file.
2. Modify the configuration file one of the following ways:
  - To allow test commands only enter: 4lw.commands.whitelist=stat, ruok, conf, isro
  - To authorize all four-character commands enter: 4lw.commands.whitelist=*
Log in to each ZooKeeper machine and run:
```
echo ruok | nc 127.0.0.1 2181▼
```
Confirm that you get a response from each ZooKeeper instance by running:
```
imok▼
```
- If no response appears or a broken pipe error appears, then ZooKeeper is not running.
Obtain more information about the status of ZooKeeper by logging into each ZooKeeper machine and running:
```
echo status | nc 127.0.0.1 2181▼
```