Symptoms
Users may experience one or more of the following when SSH agents fail to appear online in the CloudBees CI controller UI:
-
Agent SSH logs show
Agent successfully connected and onlinefrom the agent side, but the agent remains offline in the controller UI. -
Newly named agents connect successfully, but previously existing agent names fail to come online.
-
Controller logs contain
java.nio.file.FileSystemException: Operation not permittedduring agent launch. -
Restarting the controller or deleting and recreating the agent node configuration does not resolve the issue.
Diagnosis/Treatment
Check controller logs for SlaveComputer#kill() errors
When an agent is relaunched, CloudBees CI calls SlaveComputer#kill() to clean up the prior agent session before establishing a new connection.
If this cleanup step fails, the launch is aborted before the SSH connection is even attempted, which explains why the agent-side logs show a successful connection while the controller never registers it as online.
-
Retrieve the agent-specific log from the controller. Navigate to , select the affected agent, then click Log.
-
Look for errors similar to the following near the start of the most recent launch attempt:
java.nio.file.FileSystemException: /var/jenkins_home/logs/slaves/<agentname>: Operation not permitted at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ... hudson.remoting.Channel$3 run SEVERE: Unexpected termination of the channel -
If this error is present, the root cause is a filesystem permission problem on the
$JENKINS_HOME/logs/slaves/<agentname>/directory on the controller host. Proceed with the checks below.
Check directory ownership
The logs/slaves/ directory and its subdirectories must be owned by the OS user running the CloudBees CI controller process.
-
On the controller host, check the ownership of the affected agent log directory:
ls -la $JENKINS_HOME/logs/slaves/ -
Verify the owner matches the user running the Jenkins process (for example,
jenkins). -
If the ownership is incorrect, restore it with:
chown -R jenkins:jenkins $JENKINS_HOME/logs/slaves/ -
Retry the agent launch and confirm it appears online in the controller UI.
Check for immutable file attributes
On Linux systems, the chattr utility can mark files or directories as immutable, preventing any modifications even by the owning user.
-
Check for immutable flags on the agent log directory:
lsattr -R $JENKINS_HOME/logs/slaves/<agentname>/ -
If any entries show the
iflag (for example,----i--------), remove the immutable attribute:chattr -R -i $JENKINS_HOME/logs/slaves/<agentname>/ -
Retry the agent launch.
lsattr and chattr are not supported on all filesystems and may return Operation not supported on some systems. On XFS, for example, use xfs_io -c "lsattr" and xfs_io -c "chattr -i" as equivalents.
|
Check SELinux context
On RHEL or CentOS systems with SELinux enforcing, incorrect file contexts can cause permission denials even when POSIX ownership is correct.
-
Check whether SELinux is enforcing:
getenforce -
If the output is
Enforcing, check the security context of the directory:ls -laZ $JENKINS_HOME/logs/slaves/ -
If the context appears incorrect, restore the default context:
restorecon -Rv $JENKINS_HOME/logs/slaves/ -
Retry the agent launch.
To confirm the permission fix is the root cause, try connecting an agent using a brand-new name that has no existing entry under $JENKINS_HOME/logs/slaves/. If the new agent comes online successfully while the original name does not, the issue is confirmed to be in the existing log directory for that agent name.
|