Best practices for running Jenkins container pipelines with rootless Podman on RHEL 9

Overview

Migrating a CloudBees CI agent host from RHEL 8 with rootful Docker to RHEL 9 with rootless Podman changes how container processes map to host users. Pipelines that worked under rootful Docker commonly fail after this migration with container startup errors, bind-mount permission failures, or lchown / user-namespace errors during image pull, even though the pipeline code itself has not changed.

This article covers configuration practices that prevent these failures when running Docker or Podman-based container steps (sh 'docker run …', withDockerContainer, or direct podman calls) on a CloudBees CI agent running RHEL 9 with rootless Podman.

Recommendations

Set userns=keep-id as the default for the agent’s service account

Configure the CloudBees CI agent’s service account to default to keep-id user namespace mode, rather than adding --userns=keep-id to individual pipeline steps.

Explanation:

Rootless Podman’s default namespace mode maps the container’s UID to a different, unprivileged UID range on the host (the subuid/subgid offset), not to the same UID as the host process. A container started with --user <uid>:<gid> under the default mode cannot write to bind-mounted directories owned by that same UID on the host, because inside the user namespace it is a different UID entirely. keep-id mode maps the host user’s UID/GID to the same UID/GID inside the container, so container processes can read and write bind-mounted files they already own on the host.
Setting this once covers every pipeline on the node, including steps that construct their own docker/podman invocation internally (such as withDockerContainer), without requiring per-pipeline flags.

Set it in the service account’s Podman configuration:

# ~/.config/containers/containers.conf, for the Jenkins agent's service account
[containers]
userns = "keep-id"▼

This file must be created under the home directory of the OS account the CloudBees CI agent process actually runs as, not the account used to connect interactively.

Never hardcode a numeric --user flag in pipeline container steps

Replace any hardcoded --user <uid>:<gid> argument in Pipeline code with a dynamic lookup of the agent’s current identity.

Explanation:

A numeric UID that was correct under rootful Docker on RHEL 8 is not guaranteed to resolve to the same effective permissions under rootless Podman on RHEL 9, because of the user-namespace mapping described above.
A hardcoded UID also breaks silently if the service account’s UID ever changes (for example, after resolving a UID collision), since the pipeline keeps referencing the old, now-incorrect value.

Example:

// Avoid: hardcoded UID/GID, breaks under a different namespace mapping or after any UID change
sh 'docker run --rm --user 1001:1001 -v "$WORKSPACE:/workspace" myimage:latest mvn -f /workspace/pom.xml compile'

// Use: resolves dynamically to the agent's actual current identity
sh 'docker run --rm --user $(id -u):$(id -g) -v "$WORKSPACE:/workspace" myimage:latest mvn -f /workspace/pom.xml compile'▼

For a bind-mounted volume that still needs its ownership adjusted at container start, add the :U mount flag (for example -v /tmp/m2-repo:/source/repository:z,U) so Podman chowns the mount to the container’s effective user.

Verify subuid/subgid ranges exist for the service account before relying on rootless containers

Confirm that /etc/subuid and /etc/subgid both contain an entry for the CloudBees CI agent’s service account before troubleshooting further container failures.

Explanation:

Rootless Podman requires a range of subordinate UIDs and GIDs allocated to the running user in order to construct the user namespace used for pulling and running container images.
Without an entry in both files, image extraction fails with errors such as potentially insufficient UIDs or GIDs available in user namespace and lchown: invalid argument, independent of any userns or --user configuration.

grep jenkins-agent /etc/subuid /etc/subgid▼

Expected output shows a matching range in both files:

/etc/subuid:jenkins-agent:165536:2000000
/etc/subgid:jenkins-agent:165536:2000000▼

If an entry is missing from one file but present in the other, add the missing line rather than assuming both files stay in sync automatically.

Reapply ownership and SELinux context across the entire home directory after changing a service account’s UID

After running usermod -u on the CloudBees CI agent’s service account, reapply ownership and SELinux labels recursively across its home directory and workspace, not just the top-level directory.

Explanation:

Files and subdirectories created while the account ran under its previous UID keep the old numeric ownership after the UID changes, even though ls may appear to show the expected username due to NSS caching or stale listings.
On RHEL 9, sshd’s `StrictModes setting rejects public-key authentication if the home directory or .ssh contents are not owned by the authenticating user, and SELinux denies access to files with an incorrect or missing context, even once ownership is corrected.
A partial fix (chown on the home directory alone) leaves subdirectories such as .config owned by the old UID, which later surfaces as unrelated-looking failures, for example Podman refusing to write login credentials because .config is "not owned by the current user."

chown -R jenkins-agent:jenkins-agent /home/jenkins-agent
restorecon -Rv /home/jenkins-agent▼

Run this immediately after the usermod command, in the same maintenance window, rather than waiting for a failure to appear. A delay only means the inconsistency surfaces later, in a less obvious place.

Implementation

Verifying the configuration after applying these practices

Reconnect the CloudBees CI agent so the OS session picks up any account changes.
Run a pipeline step directly on the affected agent to confirm its effective identity and container runtime directory:
```
node('rhel9-agent') {
    sh 'id && echo XDG=$XDG_RUNTIME_DIR'
}▼
```
Confirm the XDG_RUNTIME_DIR value matches the service account’s actual UID, for example /run/user/1001, not the UID of an unrelated local account.
Run an existing container-based pipeline stage that writes to a bind-mounted workspace or Maven repository directory, and confirm it completes without permission or user-namespace errors.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.