Cluster Environment: Amazon Web Services

CloudBees will no longer be supporting CloudBees Jenkins Enterprise 1.x after July 30, 2020. This end-of-life announcement allows CloudBees to focus on driving new technology and product innovation for CloudBees CI. For information on moving to CloudBees CI, please refer to Migrating from CloudBees Jenkins Enterprise 1.x to CloudBees CI on modern cloud platforms which has been created to help you with the migration process. Existing customers can also contact their CSM to help ensure a smooth transition.

CloudBees Jenkins Enterprise works with all of the Amazon Elastic Compute Cloud (Amazon EC2) instance types, although some of them only work as designed in conjunction with a virtual private cloud.

AWS Credentials

There are two ways to configure an AWS environment: with or without support for EBS Storage. Regardless of which way you choose, AWS configuration requires two different sets of credentials.

First, you will need a set of credentials that will be used to setup the CloudBees Jenkins Enterprise environment. We recommend creating an IAM user with a dedicated policy that will allow for the actions needed in order to create the environment.

An example IAM policy you may use for this user is shown below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "NotAction": [
        "iam:*",
        "sts:*"
      ],
      "Resource": "*"
    }
  ]
}

Second, you will need a set of credentials that will be used for the worker process that is responsible for volume and snapshot management. There are two ways to furnish these credentials.

By default, the same set of credentials used above will also be used for this service. These credentials will be passed into the container at install time and used for the lifetime of the service. No extra configuration needs to be performed in order to use this configuration.

Using IAM Roles

Another way to achieve this is to specify an instance profile to be used by the worker instances. This is specified in the $PROJECT/cluster-init.config file in the [aws] section alongside the above credentials. The variable name is worker_instance_profile.

In order to use this instance profile, you must have created a role in the IAM settings. The role must have a trust policy defined that allows it to be launched via the EC2 service. For example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

In addition, because the credentialed user above will start the worker instance with this role, it must explicitly have the iam:PassRole ability, which makes our initial IAM policy a bit longer:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "NotAction": [
                "iam:*",
                "sts:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::123412341234:role/yourrole"
        }
    ]
}
If you choose to use IAM Roles you will need to add an entry in your .aws/config, see the next section on AWS Credentials for more information.

Now the worker process will not have hard-coded credentials, but instead will use the EC2 metadata service to get a temporary set of credentials.

These credentials will be used to manage: * lifecycle of EBS volumes and snapshots * data on S3 buckets

Last, this worker role needs to have a policy stating what kinds of AWS operations it can perform. A minimum set of AWS operations in policy form you can apply is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:CreateTags",
                "ec2:CreateSnapshot",
                "ec2:CreateVolume",
                "ec2:AttachVolume",
                "ec2:DeleteVolume",
                "ec2:DetachVolume",
                "ec2:DeleteSnapshot",
                "ec2:DescribeSnapshots",
                "ec2:DescribeVolumes",
                "ec2:DescribeInstances",
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersionAcl",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload",
                "s3:RestoreObject"
            ],
            "Resource": [
                "arn:aws:s3:::cje-*"
            ],
            "Effect": "Allow"
        }
    ]
}

cje-* needs to be adjusted to match <cluster-name>-*.

The ecr actions are only needed if an EC2 Container Registry is used.

For help managing AWS permissions refer to AWS - Manage IAM permissions.

AWS Credential Storage

Prior to version 1.5.0, AWS credentials were stored in cluster-init.secrets.

Credentials may continue to be stored and used from within this file. However CloudBees Jenkins Enterprise now contains support for using credentials that are stored within the conventional AWS tool credential locations. By default this file is ~/.aws/credentials, but can be overwritten with the AWS_CREDENTIAL_FILE environment variable.

In order to use credentials stored in the shared credentials file, you must specify the credential_profile in the cluster-init.config.

If you are using AWS IAM Roles

CloudBees Jenkins Enterprise allows you to use Roles Terms and Concepts to control access to AWS. By configuring role assumption parameters, and specifying the role profile name in the credential_profile config entry, CloudBees Jenkins Enterprise will attempt to change roles before performing AWS operations. This can allow you to provide a shared role for CloudBees Jenkins Enterprise administration access while handing out more limited capability credentials directly to end CloudBees Jenkins Enterprise administrators.

When making use of the credential_profile option vs. using the cluster-init.secrets, the worker_instance_profile must be specified and used. System credentials will not be copied onto worker instances.

You must also add a configuration for your profile to your .aws/config file, for example here is an .aws/config file that specifies a profile with the name "my-developer-profile". Make sure the information here matches what you setup in AWS IAM for your Role & Profile.

[default]
region = us-east-1

[profile my-developer-profile]
region=us-east-1
role_arn = arn:aws:iam::9999998999:role/developer
source_profile = my-company-iam
mfa_serial = arn:aws:iam::974808099065:mfa/joesmith
role_session_name = joesmith-developer
The CloudBees Jenkins Enterprise cje tool uses the Boto3 Python Library to access and control AWS. For more information on Boto’s support for AWS IAM Roles see: Boto docs: AWS Assume Role Provider. For more information on how AWS credentials must be configured for Boto see: Boto docs: Shared Credentials.

EBS Usage

If you are planning to use EBS (Elastic Block Store) as storage, the AWS credentials used for installation must also be able to create EBS resources.

AWS Components

CloudBees Jenkins Enterprise in an AWS environment uses the following Amazon Web Services:

  • EC2 (Elastic Compute Cloud) to provide resizable compute capacity

  • S3 (Simple Storage Services) to store data objects in resources called "buckets"

  • ELB (Elastic Load Balancing) to redirect data traffic as needed

  • EBS (Elastic Block Store) to provide persistent block-level storage volumes for use with Amazon EC2 instances in the AWS Cloud

  • Route 53 (domain name system) for translating Web URL names into numeric IP addresses

  • VPC (virtual private cloud) to enable provisioning of a logically isolated section of AWS

Although some clusters do not use Route 53 or VPC, we recommend that configuration support those components anyway.

The recommended configuration in an AWS environment enables three different types of workers (virtual machines) as described in the following table:

General Purpose

Operations Center and Jenkins Masters run as containerized processes on general-purpose workers.

Recommended instance type: M4 2XL (32 GiB)

Dedicated ES

Elasticsearch is performed by dedicated ES workers.

Recommended instance type: R3 XL (30.5 GiB)

Executors

Jenkins agents run on executors.

Recommended instance type: varies by intended workload

The I/O performance of individual instance types differs between OpenStack and AWS.

Recommended Executor instance types are as follows: - M4.2xlarge is best for basic web applications and generic workloads - C3.xlarge is best for C and C++ codebases and complex builds - i2.xlarge is best for I/O-intensive builds such as those associated with building large indexes

Deployments can be thought of as small, medium, or large depending on the number of Jenkins masters that are running (up to 3, up to 10, and up to 30, respectively).

Guidelines for resource allocation across deployments of different size are as follows (The recommended instance for all controllers is M4.large. Controllers route requests and run Apache Mesos to support resource provisioning):

Small

3 controllers and 3 General Purpose workers

Medium

3 controllers, 3 ES workers, 5 General Purpose workers

Large

3 controllers, 5 ES workers, 5 General Purpose workers, 8 Executors