Managed Masters in specific Kubernetes namespaces

Managed masters are by default created in the same namespace as the Operations Center instance. If you want to create a managed master in a different namespace, you need to pre-populate the namespace with the appropriate resources.

Adding a specific Namespace with Helm

Adding a specific Namespace using Helm

The process for creating a new namespace using Helm (with or without Tiller) is:

echo "Enter namespace to set up:"
read NAMESPACE
cat << EOF > values.yaml
OperationsCenter:
    Enabled: false
Master:
    Enabled: true
    OperationsCenterNamespace: $NAMESPACE
Agents:
    Enabled: true
EOF
kubectl create namespace $NAMESPACE || true
helm fetch cloudbees/cloudbees-core --untar
helm template cloudbees-core --name cloudbees-core-masters-$NAMESPACE --namespace $NAMESPACE -f values.yaml | kubectl apply -f -

Adding a specific Namespace manually

Adding a specific namespace manually

The process for creating a new namespace is:

  1. Using a terminal with access to the Kubernetes instance running Operations Center, create the namespace for the new master:

    shell% kubectl create namespace my-namespace
  2. Create a file called setup_namespace.sh and populate it with:

    The names of resources have changed since versions 2.176.2.3 of CloudBees Core and the introduction of the cloudbees/cloudbees-core helm chart. For those older versions:

    • Replace cjoc-agents with pods-all

    • Replace cjoc-master-role-binding with jenkins

    • Replace cjoc-master-management with master-management

    • Replace cjoc-role-binding with`cjoc`

    #!/bin/bash
    
    echo "Enter namespace that CJOC is running in:"
    read cjoc_namespace
    
    echo "Enter namespace for new Master:"
    read new_master_namespace
    
    kubectl get ServiceAccount jenkins -n ${cjoc_namespace} -o yaml > output.yml
    echo "---" >> output.yml
    kubectl get RoleBinding cjoc-master-role-binding -n ${cjoc_namespace} -o yaml >> output.yml
    echo "---" >> output.yml
    kubectl get ConfigMap jenkins-agent -n ${cjoc_namespace} -o yaml >> output.yml
    echo "---" >> output.yml
    kubectl get Role cjoc-agents -n ${cjoc_namespace} -o yaml >> output.yml
    echo "---" >> output.yml
    kubectl get Role cjoc-master-management -n ${cjoc_namespace} -o yaml >> output.yml
    kubectl get RoleBinding cjoc-role-binding -n ${cjoc_namespace} -o yaml > rolebinding.yml
    
    sed -i '' 's/namespace: '${cjoc_namespace}'/namespace: '${new_master_namespace}'/g;s/"namespace":"'${cjoc_namespace}'"/"namespace":"'${new_master_namespace}'"/g' output.yml
    sed -i '' 's/namespace: '${cjoc_namespace}'/namespace: '${new_master_namespace}'/g;s/"namespace":"'${cjoc_namespace}'"/"namespace":"'${new_master_namespace}'"/g' rolebinding.yml
  3. Execute the setup_namespace script. The script prompts you for the namespace on which Operations Center is running and the desired namespace for the new master, and generates two files (output.yml and rolebinding.yml):

    shell% ./setup_namespace.sh
    Enter namespace that CJOC is running in:
    cjoc-namespace
    Enter namespace for new Master:
    my-namespace
  4. (OPTIONAL) Review the output.yml file and edit or remove the following values, as needed:

    kubectl.kubernetes.io/last-applied-configuration
    creationTimestamp
    resourceVersion
    selfLink
    secrets
    uid
  5. Edit the rolebinding.yml file (which links the Operations Center account to the new namespace) and review the subjects: section. If the namespace value is present, make sure it matches the Operations Center namespace, and if the namespace value is missing, add a namespace: line to the subjects: block:

    subjects:
    - kind: ServiceAccount
      name: cjoc
      namespace: ${name-of-operations-center-namespace}
  6. Apply the modified output.yml and rolebinding.yml files to the Kubernetes cluster:

    shell% kubectl apply -f ${path-to-output.yml}
    shell% kubectl apply -f ${path-to-rolebinding.yml}
  7. Using Operations Center, start the new master.

Managed Masters in specific OpenShift projects

By default, managed masters are created in the same projects that Operations Center is running in.

To create a managed master in a specific OpenShift project, the project must be pre-created with the proper resources.

Those resources are:

  • The 'jenkins' ServiceAccount that will be used by the managed master(s) to provision Jenkins agents.

  • The Role and RoleBinding of the 'jenkins' ServiceAccount

  • The Role and RoleBinding of Operations Center ServiceAccount to allow Operations Center to manage the master resources

Red Hat recommends that OpenShift production clusters use the ovs-multitenant network plugin. This plugin makes it so no namespaces can reference each others services without going through a route exposed on the router.

If ovs-multitentant is enabled, then the project running Operations Center needs to be a global project to run managed masters in other projects. Use the oc adm command below to make the project global; replace cloudbees with the name of your project.

oc adm pod-network make-projects-global cloudbees

Here is the definition of the 'jenkins' service account and associated Role and RoleBinding:

The RoleBinding namespace '<PROJECT-MASTER-X>' should be the newly created project name.
apiVersion: v1
kind: List
items:
 -
  kind: ServiceAccount
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: jenkins

 -
  kind: Role
  apiVersion: v1
  metadata:
    name: pods-all
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]

 -
  kind: RoleBinding
  apiVersion: v1
  metadata:
    name: jenkins
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: pods-all
    # The new project name
    namespace: <PROJECT-MASTER-X>
  subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: <PROJECT-MASTER-X>

To create a managed master in a specific OpenShift project, Operations Center must have the Role privileges to do so.

The RoleBinding namespace '<PROJECT-MASTER-X>' should be the newly created project name.

The RoleBinding must specify the namespace in which the cjoc ServiceAccount is defined (in the following example, cje).

apiVersion: v1
kind: List
items:
 -
  kind: Role
  apiVersion: v1
  metadata:
    name: master-management
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: ["route.openshift.io",""]
    resources: ["routes"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: ["route.openshift.io"]
    resources: ["routes/custom-host"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["list"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get","list","watch"]

 -
  kind: RoleBinding
  apiVersion: v1
  metadata:
    name: cjoc
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: master-management
    namespace: <PROJECT-MASTER-X>
  subjects:
  - kind: ServiceAccount
    name: cjoc
    # cjoc service account project name
    namespace: cje

Optionally, you can give Operations Center the privileges to list namespaces so that the user can select the project/namespace instead of typing the namespace in. To accomplish this, Operations Center must have the ClusterRole privileges to do so.

The ClusterRoleBinding must specify the namespace in which the cjoc ServiceAccount is defined (in the following example, cje).
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cjoc-ns-management
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["list"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cjoc-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cjoc-ns-management
subjects:
- kind: ServiceAccount
  name: cjoc
  # cjoc service account namespace
  namespace: cje

Master provisioning configuration

To provision masters in their own projects, each master must use a specific sub-domain. For example, if the Operations Center domain is 'cd.example.org' and the URL is 'https://cb.example.org/cjoc/', a master dev1 is needed to use the sub-domain 'dev1.cd.example.org' or 'dev1-cd.example.org'. It is often preferable to use the latter if using a wild card certificates for domain 'example.org'.

To configure each master to use a specific sub-domain, set the 'Master URL Pattern' in the main Jenkins configuration page 'Manage Jenkins → Configure System' under 'Kubernetes Master Provisioning' advanced options. For example if the Operations Center domain is 'cd.example.org', the 'Master URL Pattern' would be 'https://*-cd.example.org/*/'.

Provision Masters

The project for the master resources can be configured as the default project for all managed masters in the main Operations Center configuration screen with the 'namespace' parameter.

The project can also be specify for a specific managed master in the master configuration screen with the 'namespace' parameter.

Leave the namespace value empty to use the value defined by the Kubernetes endpoint.

Managing masters with Operations Center

Managing Masters

When new teams join an organization, or existing teams start a new project, CloudBees Core makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees Core, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees Core environment can handle is limited only by the capacity of the cluster.

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees Core administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk upgrades

When a CloudBees Core cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees Core, this can be achieved by defining a cluster operation in Operations Center.

To create a task to perform a bulk upgrade:

  1. Log into Operations Center as an administrator.

  2. Create a New Item for Cluster Operations.

  3. Select the Managed Masters cluster operation

  4. Select the pre-configured upgrade pattern that you want to use.

  5. In the Uses Docker Image, pick a Docker image that is used by the cluster’s masters as the upgrade target. Any master in the cluster using the selected image will be affected by this cluster operation.

  6. In the Steps section, select Update Docker Image and then pick the new Docker to which you want to bulk-upgrade the targeted masters.

  7. Add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Operating Managed and Client Masters

When new teams join an organization, or existing teams start a new project, CloudBees Core makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees Core, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees Core environment can handle is limited only by the capacity of the cluster.

Adding Client Masters

Occasionally administrators will need to connect existing masters to a CloudBees Core cluster, such as in the case of a team requiring a master on Windows. Existing masters that are connected to Operations Center lack key benefits of Managed Masters like high availability and automatic agent management. Whenever possible, administrators should use a Managed Master with CloudBees Core rather than connecting an existing master.

Client Masters are monitored by Operations Center just as Managed Masters are monitored. Administrators can see the status of all their Managed Masters, Team Masters, and Client Masters from the Operations Center masters page. Client Masters can receive configuration updates from Operations Center with configuration snippets. Client Masters can share agents hosted in the cluster, offloading the burden of agent management from teams.

Client Masters do not have the high availability features of Managed Masters.

Note: The existing Client Master and Operations Center must both accept JNLP requests.

To add a Client Master, log into CloudBees Core and navigate to the Operations Center dashboard. Click the New Item option in the left-hand menu and provide the following information:

  • Item name: the name of the existing Client Master to connect to the cluster

  • Select Client Master. Just as with Managed Masters, Client Masters offer some customization and configuration options on creation:

  • On-master executors: Number of builds to execute concurrently on the Client Master itself. Default setting is 2.

  • Email addresses: Contact information for the administrator responsible for maintaining the Client Master.

Once these settings are saved, Operations Center will attempt to connect the Client Master to the CloudBees Core cluster.

Verify that Operations Center and the existing Client Master can communicate with each other over both HTTP and JNLP ports. Host and port to use for JNLP is advertised through HTTP headers by each Jenkins master.

You can connect an existing Client Master to Operations Center by giving that Client Master the TLS certificate for Operations Center, typically through the Configure Global Security page in Operations Center. For more information, see How to programmatically connect a Client Master to CJOC.

If you are connecting multiple Client Masters to your cluster, it is a good idea to automate that task using shared configurations.

Once the Client Master is connected, administrators should configure security and access controls for the master.

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees Core administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk-upgrading Managed Masters

When a CloudBees Core cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees Core, this can be achieved by defining a cluster operation in Operations Center.

To create this task, the administrator will first need to log into Operations Center, then create a New Item of the type Cluster Operations. The administrator will then needs to select the Managed Masters cluster operation, and will then be given a set of pre-configured upgrade patterns to choose from.

The administrator will then need to specify which masters to target by using the filter Uses Docker Image and picking a Docker image used by their cluster’s masters as the upgrade target for this operation. Any masters in the cluster using the selected image will be affected by this cluster operation.

In the Steps section, the administrator should select Update Docker Image and pick the new Docker image to bulk-upgrade the targeted masters to. Next, the administrator should add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Quiet start

There may be times during an upgrade or other maintenance when it is best to have Jenkins start, but launch no projects. For example, if an upgrade is being performed in multiple steps, the intermediate steps may not be fully configured to run projects successfully. The "quiet start plugin" can immediately place the Jenkins server in "quieting down" state on startup.

Enable "quiet start" by checking the box in Manage Jenkins  Quiet Restart. When the server is restarted, it will be in the "quieting down" state. An administrator can cancel that state using the regular UI.

Uncheck the box in Manage Jenkins  Quiet Restart when maintenance is complete. Projects will start as usual on server restart.

Reviewing plugin usage

Large Jenkins installations often have many plugins installed. CloudBees Core can help administrators identify plugins which may be unused or no longer useful. The CloudBees Plugin Usage Plugin helps you keep track of which plugins you are actively using and where they are used.

Plugin usage report

A table of installed plugins is available from Manage Jenkins  Plugin Usage, including a usage count and a list of uses of that plugin. A strikethrough font indicates disabled plugins. A progress bar runs while Jenkins scans jobs and global settings. Once the scan is complete, the Usage Count column becomes a sortable column. Click the column heading to sort the column.

usage count
Figure 1. Usage Count

The third column shows detected uses of the plugin, hyperlinked to the matching configuration screen. In the list of hyperlinks, "Jenkins" refers to global Jenkins configuration. Most other labels are names of configurable items such as jobs (using » to show folder structure where relevant); configuration associated with Jenkins users (such as credentials) will also be shown. Code packaged as a plugin but really used only as a library shared between several "real" plugins (Async Http Client Plugin, for example) will be shown as used by those plugins. »… is appended to items which have nested elements also using the plugin; for example, a native Maven project may have several modules all of which refer to the plugin.

Only a Jenkins administrator can perform a complete plugin usage analysis. Configuration read permission is required to check plugin usage.

Limitations

Beware that some plugins will not be listed in this table, because they are not directly mentioned in any stored configuration. They may affect how Jenkins runs in various ways without configuration; the Jenkins Translation Assistance Plugin is an example.

Conversely, a plugin might have some stored configuration which is of no interest or never used at all. For example, Jenkins Email Extension Plugin displays various controls in the global Configure System screen; when you click Save, Email Extension configuration is saved in hudson.plugins.emailext.ExtendedEmailPublisher.xml in your $JENKINS_HOME, even if you have made no customizations. This plugin will thus appear to be "in use" by Jenkins. Only if you have enabled the Editable Email Notification post-build action for a job will it have a usage count greater than 1, however.

Use the plugin usage table as a starting point for investigating plugins that might be unused. It is not relevant for all plugins. The table works best for checking the usage of plugins which offer concrete additions to configuration, such as build steps or publishers in jobs.

Plugins used in configuration generated by a builder (or publisher) template will not be listed (in either the template or jobs using it), since this configuration is created on the fly during a build rather than being stored permanently in the job. Configuration generated by a job (or folder) template is considered.

Copyright © 2010-2019 CloudBees, Inc.Online version published by CloudBees, Inc. under the Creative Commons Attribution-ShareAlike 4.0 license.CloudBees and CloudBees DevOptics are registered trademarks and CloudBees Core, CloudBees CodeShip, CloudBees Jenkins Enterprise, CloudBees Jenkins Platform, CloudBees Jenkins Operations Center and DEV@cloud are trademarks of CloudBees, Inc. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Jenkins is a registered trademark of the non-profit Software in the Public Interest organization. Used with permission. See here for more info about the Jenkins project. The registered trademark Jenkins® is used pursuant to a sublicense from the Jenkins project and Software in the Public Interest, Inc. Read more at www.cloudbees.com/jenkins/about. Apache, Apache Ant, Apache Maven, Ant and Maven are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.Other names may be trademarks of their respective owners. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this content, and CloudBees was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this content, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.