Why is my instance not online after a server restart?

2 minute readKnowledge base

Issue

After I restarted my server, my CloudBees CI instance did not start up automatically. After logging into the server and manually starting the CloudBees CI service, my instance is back online.

Environment

Resolution

First check if the service is enabled for automatic startups:

[root@hostname ~]# systemctl is-enabled cloudbees-core-cm
enabled

If your JENKINS_HOME is located on a NFS mount, it is possible that after a server restart that your NFS mount is not available at the time when your CloudBees CI instance service is initializing.

In this journalctl output, we see that the NFS mount is not available yet.

[root@hostname ~]# journalctl -u cloudbees-core-cm
-- Logs begin at Wed 2023-03-09 12:10:48 PST, end at Wed 2023-03-09 12:11:22 PST. --
Mar 09 12:10:51 hostname systemd[1]: Started cloudbees-core-cm.
Mar 09 12:10:51 hostname cloudbees-core-cm[740]: JENKINS_HOME directory does not exist: /nfsmount/cloudbees-core-cm
Mar 09 12:10:51 hostname systemd[1]: cloudbees-core-cm.service: main process exited, code=exited, status=1/FAILURE
Mar 09 12:10:51 hostname systemd[1]: Unit cloudbees-core-cm.service entered failed state.
Mar 09 12:10:51 hostname systemd[1]: cloudbees-core-cm.service failed.

However, users are still able to reach the mount after logging into the server.

[root@hostname ~]$ ls /nfsmount/cloudbees-core-cm/ -la
total 492
drwxr-xr-x.  28 cloudbees-core-cm cloudbees-core-cm  8192 Mar 10 05:46 .
drwxrwxrwx.   4 root              root                 60 Mar  8 11:21 ..
drwxr-xr-x.   4 cloudbees-core-cm cloudbees-core-cm    70 Mar  8 11:21 caches
drwxr-xr-x.  15 cloudbees-core-cm cloudbees-core-cm  4096 Mar 10 05:41 cb-envelope
...

In this case, the service started before the NFS mount was available, and if you try to start the service manually, it will come online sucessfully, since NFS is now mounted.

This issue is fixed in release CloudBees CI on traditional platforms 2.387.3.3 tracked by BEE-32236.

Workaround

You can add additional Wants and After settings to the systemd service drop-in file as per Overriding vendor settings. For example using sudo systemctl edit cloudbees-core-cm, adding the following lines:

[Unit]
Wants=network-online.target
Wants=remote-fs.target
After=network-online.target
After=remote-fs.target

After saving, you should see the following override file created:

cat /etc/systemd/system/cloudbees-core-cm.service.d/override.conf
[Unit]
Wants=network-online.target
Wants=remote-fs.target
After=network-online.target
After=remote-fs.target