Managed Masters in specific Kubernetes namespaces

Managed Masters are by default created in the same namespace as the Operations Center instance. If you want to create a Managed Master in a different namespace, you need to pre-populate the namespace with the appropriate resources.

Adding a specific Namespace with Helm

Adding a specific Namespace using Helm

The process for creating a new namespace using Helm is:

echo "Enter namespace where Operations Center is deployed:"
read OC_NAMESPACE
echo "Enter namespace to set up:"
read NAMESPACE
cat << EOF > values.yaml
OperationsCenter:
    Enabled: false
Master:
    Enabled: true
    OperationsCenterNamespace: $OC_NAMESPACE
Agents:
    Enabled: true
EOF
kubectl create namespace $NAMESPACE || true
helm fetch cloudbees/cloudbees-core --untar
helm template cloudbees-core-masters-$NAMESPACE cloudbees-core --namespace $NAMESPACE -f values.yaml | kubectl apply -n $NAMESPACE -f -

Adding a specific Namespace manually

Adding a specific namespace manually

The process for creating a new namespace is:

  1. Using a terminal with access to the Kubernetes instance running Operations Center, create the namespace for the new master:

    shell% kubectl create namespace my-namespace
  2. Create a file called setup_namespace.sh and populate it with:

    The names of resources have changed since versions 2.176.2.3 of CloudBees CI and the introduction of the cloudbees/cloudbees-core helm chart. For those older versions:

    • Replace cjoc-agents with pods-all

    • Replace cjoc-master-role-binding with jenkins

    • Replace cjoc-master-management with master-management

    • Replace cjoc-role-binding with`cjoc`

    #!/bin/bash
    
    echo "Enter namespace that CJOC is running in:"
    read cjoc_namespace
    
    echo "Enter namespace for new Master:"
    read new_master_namespace
    
    kubectl get ServiceAccount jenkins -n ${cjoc_namespace} -o yaml > output.yml
    echo "---" >> output.yml
    kubectl get RoleBinding cjoc-master-role-binding -n ${cjoc_namespace} -o yaml >> output.yml
    echo "---" >> output.yml
    kubectl get ConfigMap jenkins-agent -n ${cjoc_namespace} -o yaml >> output.yml
    echo "---" >> output.yml
    kubectl get Role cjoc-agents -n ${cjoc_namespace} -o yaml >> output.yml
    echo "---" >> output.yml
    kubectl get Role cjoc-master-management -n ${cjoc_namespace} -o yaml >> output.yml
    kubectl get RoleBinding cjoc-role-binding -n ${cjoc_namespace} -o yaml > rolebinding.yml
    
    sed -i '' 's/namespace: '${cjoc_namespace}'/namespace: '${new_master_namespace}'/g;s/"namespace":"'${cjoc_namespace}'"/"namespace":"'${new_master_namespace}'"/g' output.yml
    sed -i '' 's/namespace: '${cjoc_namespace}'/namespace: '${new_master_namespace}'/g;s/"namespace":"'${cjoc_namespace}'"/"namespace":"'${new_master_namespace}'"/g' rolebinding.yml
  3. Execute the setup_namespace script. The script prompts you for the namespace on which Operations Center is running and the desired namespace for the new master, and generates two files (output.yml and rolebinding.yml):

    shell% ./setup_namespace.sh
    Enter namespace that CJOC is running in:
    cjoc-namespace
    Enter namespace for new Master:
    my-namespace
  4. (OPTIONAL) Review the output.yml file and edit or remove the following values, as needed:

    kubectl.kubernetes.io/last-applied-configuration
    creationTimestamp
    resourceVersion
    selfLink
    secrets
    uid
  5. Edit the rolebinding.yml file (which links the Operations Center account to the new namespace) and review the subjects: section. If the namespace value is present, make sure it matches the Operations Center namespace, and if the namespace value is missing, add a namespace: line to the subjects: block:

    subjects:
    - kind: ServiceAccount
      name: cjoc
      namespace: ${name-of-operations-center-namespace}
  6. Apply the modified output.yml and rolebinding.yml files to the Kubernetes cluster:

    shell% kubectl apply -f ${path-to-output.yml}
    shell% kubectl apply -f ${path-to-rolebinding.yml}
  7. Using Operations Center, start the new master.

Managed Masters in specific OpenShift projects

By default, Managed Masters are created in the same projects that Operations Center is running in.

To create a Managed Master in a specific OpenShift project, the project must be pre-created with the proper resources.

Those resources are:

  • The 'jenkins' ServiceAccount that will be used by the Managed Master(s) to provision Jenkins agents.

  • The Role and RoleBinding of the 'jenkins' ServiceAccount

  • The Role and RoleBinding of Operations Center ServiceAccount to allow Operations Center to manage the master resources

Red Hat recommends that OpenShift production clusters use the ovs-multitenant network plugin. This plugin makes it so no namespaces can reference each others services without going through a route exposed on the router.

If ovs-multitentant is enabled, then the project running Operations Center needs to be a global project to run Managed Masters in other projects. Use the oc adm command below to make the project global; replace cloudbees with the name of your project.

oc adm pod-network make-projects-global cloudbees

Here is the definition of the 'jenkins' service account and associated Role and RoleBinding:

The RoleBinding namespace '<PROJECT-MASTER-X>' should be the newly created project name.
apiVersion: v1
kind: List
items:
 -
  kind: ServiceAccount
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: jenkins

 -
  kind: Role
  apiVersion: v1
  metadata:
    name: pods-all
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]

 -
  kind: RoleBinding
  apiVersion: v1
  metadata:
    name: jenkins
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: pods-all
    # The new project name
    namespace: <PROJECT-MASTER-X>
  subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: <PROJECT-MASTER-X>

To create a Managed Master in a specific OpenShift project, Operations Center must have the Role privileges to do so.

The RoleBinding namespace '<PROJECT-MASTER-X>' should be the newly created project name.

The RoleBinding must specify the namespace in which the cjoc ServiceAccount is defined (in the following example, cje).

apiVersion: v1
kind: List
items:
 -
  kind: Role
  apiVersion: v1
  metadata:
    name: master-management
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: ["route.openshift.io",""]
    resources: ["routes"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: ["route.openshift.io"]
    resources: ["routes/custom-host"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["list"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get","list","watch"]

 -
  kind: RoleBinding
  apiVersion: v1
  metadata:
    name: cjoc
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: master-management
    namespace: <PROJECT-MASTER-X>
  subjects:
  - kind: ServiceAccount
    name: cjoc
    # cjoc service account project name
    namespace: cje

Optionally, you can give Operations Center the privileges to list namespaces so that the user can select the project/namespace instead of typing the namespace in. To accomplish this, Operations Center must have the ClusterRole privileges to do so.

The ClusterRoleBinding must specify the namespace in which the cjoc ServiceAccount is defined (in the following example, cje).
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cjoc-ns-management
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["list"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cjoc-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cjoc-ns-management
subjects:
- kind: ServiceAccount
  name: cjoc
  # cjoc service account namespace
  namespace: cje

Master provisioning configuration

To provision masters in their own projects, each master must use a specific sub-domain. For example, if the Operations Center domain is 'cd.example.org' and the URL is 'https://cb.example.org/cjoc/', a master dev1 is needed to use the sub-domain 'dev1.cd.example.org' or 'dev1-cd.example.org'. It is often preferable to use the latter if using a wild card certificates for domain 'example.org'.

To configure each master to use a specific sub-domain, set the 'Master URL Pattern' in the main Jenkins configuration page 'Manage Jenkins → Configure System' under 'Kubernetes Master Provisioning' advanced options. For example if the Operations Center domain is 'cd.example.org', the 'Master URL Pattern' would be 'https://*-cd.example.org/*/'.

Provision Masters

The project for the master resources can be configured as the default project for all Managed Masters in the main Operations Center configuration screen with the 'namespace' parameter.

The project can also be specify for a specific Managed Master in the master configuration screen with the 'namespace' parameter.

Leave the namespace value empty to use the value defined by the Kubernetes endpoint.

Managing masters with Operations Center

When new teams join an organization, or existing teams start a new project, CloudBees CI makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees CI, a Master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of Masters a environment can handle is limited only by the capacity of the cluster.

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, an administrator needs to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk upgrades

When a cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. This can be achieved by defining a cluster operation in Operations Center.

To create a task to perform a bulk upgrade:

  1. Sign in to Operations Center as an administrator.

  2. Create a New Item for Cluster Operations.

  3. Select the Managed Masters cluster operation

  4. Select the pre-configured upgrade pattern that you want to use.

  5. In the Uses Docker Image, pick a Docker image that is used by the cluster’s masters as the upgrade target. Any master in the cluster using the selected image will be affected by this cluster operation.

  6. In the Steps section, select Backup Master and configure the options as per Using the CloudBees Backup plugin

  7. In the Steps section, select Update Docker Image and then pick the new Docker to which you want to bulk-upgrade the targeted masters.

  8. Add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Operating Managed and Client Masters

When new teams join an organization, or existing teams start a new project, CloudBees CI makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees CI, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees CI environment can handle is limited only by the capacity of the cluster.

Adding Client Masters

Occasionally administrators will need to connect existing masters to a CloudBees Core cluster, such as in the case of a team requiring a master on Windows. Existing masters that are connected to Operations Center lack key benefits of Managed Masters like high availability and automatic agent management. Whenever possible, administrators should use a Managed Master with CloudBees CI rather than connecting an existing master.

Client Masters are monitored by Operations Center just as Managed Masters are monitored. Administrators can see the status of all their Managed Masters, Team Masters, and Client Masters from the Operations Center masters page. Client Masters can receive configuration updates from Operations Center with configuration snippets. Client Masters can share agents hosted in the cluster, offloading the burden of agent management from teams.

Client Masters do not have the high availability features of Managed Masters.

Note: The existing Client Master and Operations Center must both accept JNLP requests.

To add a Client Master, log into CloudBees Core and navigate to the Operations Center dashboard. Click the New Item option in the left-hand menu and provide the following information:

  • Item name: the name of the existing Client Master to connect to the cluster

  • Select Client Master. Just as with Managed Masters, Client Masters offer some customization and configuration options on creation:

  • On-master executors: Number of builds to execute concurrently on the Client Master itself. Default setting is 2.

  • Email addresses: Contact information for the administrator responsible for maintaining the Client Master.

Once these settings are saved, Operations Center will attempt to connect the Client Master to the CloudBees CI cluster.

Verify that Operations Center and the existing Client Master can communicate with each other over both HTTP and JNLP ports. Host and port to use for JNLP is advertised through HTTP headers by each Jenkins master.

You can connect an existing Client Master to Operations Center by giving that Client Master the TLS certificate for Operations Center, typically through the Configure Global Security page in Operations Center. For more information, see How to programmatically connect a Client Master to CJOC.

If you are connecting multiple Client Masters to your cluster, it is a good idea to automate that task using shared configurations.

Once the Client Master is connected, administrators should configure security and access controls for the master.

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees CI administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk-upgrading Managed Masters

When a CloudBees CI cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees CI, this can be achieved by defining a cluster operation in Operations Center.

To create this task, the administrator will first need to log into Operations Center, then create a New Item of the type Cluster Operations. The administrator will then needs to select the Managed Masters cluster operation, and will then be given a set of pre-configured upgrade patterns to choose from.

The administrator will then need to specify which masters to target by using the filter Uses Docker Image and picking a Docker image used by their cluster’s masters as the upgrade target for this operation. Any masters in the cluster using the selected image will be affected by this cluster operation.

In the Steps section, the administrator should select Update Docker Image and pick the new Docker image to bulk-upgrade the targeted masters to. Next, the administrator should add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Quiet start

There may be times during an upgrade or other maintenance when it is best to have Jenkins start, but launch no projects. For example, if an upgrade is being performed in multiple steps, the intermediate steps may not be fully configured to run projects successfully. The "quiet start plugin" can immediately place the Jenkins server in "quieting down" state on startup.

Enable "quiet start" by checking the box in Manage Jenkins  Quiet Restart. When the server is restarted, it will be in the "quieting down" state. An administrator can cancel that state using the regular UI.

Uncheck the box in Manage Jenkins  Quiet Restart when maintenance is complete. Projects will start as usual on server restart.