How to build images against AWS ECR when using IAM Role for Service Accounts

Article ID:360059466171
3 minute readKnowledge base

Issue

  • I would like to build my own docker images and push them to AWS ECR

Environment

Explanation

AWS IAM roles for service accounts (IRSA) allows to bind a Kubernetes ServiceAccount to IAM Roles, that allows fine-grained authorization within AWS. When building and pushing docker images from Jenkins agent to a private registries like AWS ECR, specific kubernetes ServiceAccounts can be used to provide the necessary permissions in AWS. But attaching the ServiceAccount to agent pods is not all that is needed. Further configuration is required to make sure that the tools used (Kaniko, docker, …​) are able to infer the AWS Identify from the environment.

Resolution

The solution is applicable to DinD (but not DooD - when mounting the docker socket) or a tool like Kaniko as it relies on the AWS ECR Credentials Helper.

Pre-Requisites

AWS / Kubernetes

As explained in AWS IAM roles for service accounts (IRSA):

  • There is an IAM Role that provides sufficient permission to pull / push to AWS ECR

  • There is an ServiceAccount created in the namespace where the Agent Pods are being deployed that are associated with the IAM Role

If using Kaniko

  • Kaniko version 0.16.0 or later must be used. Note: The Kaniko image at the time of writing already contains the docker credentials helper for AWS ECR. So nothing else is needed.

If using DinD

  • The AWS ECR Credentials Helper version 0.4.0 of later is installed and available in the dind container. The following is an example of a Dockerfile that creates a dind image with the AWS ECR Credentials Helper added to the PATH:

      # Stage 0: Get credential helpers
  FROM golang:1.15
  RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
  RUN make -C /go/src/github.com/awslabs/amazon-ecr-credential-helper linux-amd64

  FROM docker:19.03.11-dind
  COPY --from=0 /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/linux-amd64/docker-credential-ecr-login /usr/local/bin/docker-credential-ecr-login

Steps

In short, the following steps are required:

  1. Attach the ServiceAccount that is associated to the IAM Role to the Agent Pod .spec.serviceAccountName

  2. Inject the environment variables AWS_EC2_METADATA_DISABLED=true and AWS_SDK_LOAD_CONFIG=true to the "tool" container in the Pod. This is specific to the behavior of the AWS ECR Credentials Helper.

  3. Add the ECR credentials helper configuration ('{"credsStore":"ecr-login"}') in the docker config file (by default in the container user HOME directory ~/.docker/config.json`)

Examples

Kaniko

Here is an overall solution based on the documentation at Using Kaniko with CloudBees Core:

pipeline {
  agent {
    kubernetes {
      yaml """
kind: Pod
metadata:
  name: kaniko
spec:
  serviceAccountName: aws-iam-ecr
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - sleep
    args:
    - 9999999
    env:
    - name: AWS_EC2_METADATA_DISABLED
      value: true
    - name: AWS_SDK_LOAD_CONFIG
      value: true
"""
    }
  }
  stages {
    stage('Build with Kaniko') {
      steps {
        container(name: 'kaniko', shell: '/busybox/sh') {
          sh '''#!/busybox/sh
            dockerConfig=\${DOCKER_CONFIG:-/kaniko/.docker}
            [ -d \${dockerConfig} ] && echo "Docker directory Exists" || mkdir -p \${dockerConfig}
            echo '{"credsStore":"ecr-login"}' > \${dockerConfig}/config.json
          '''
          sh '''#!/busybox/sh
            echo "FROM jenkins/inbound-agent:latest" > Dockerfile
            /kaniko/executor --context `pwd` --destination 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hello-kaniko:latest
          '''
        }
      }
    }
  }
}
▼
DinD

Here is an overall solution based on the documentation at How to build my own docker images in CloudBees CI (CloudBees Core) on Modern Cloud Platforms:

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  serviceAccountName: aws-iam-ecr
  containers:
  - name: dind
    # `dind` image with ECR helper
    image: my-dind-with-ecr:19.03.11
    imagePullPolicy: Always
    env:
    - name: AWS_EC2_METADATA_DISABLED
      value: true
    - name: AWS_SDK_LOAD_CONFIG
      value: true
    tty: true
    securityContext:
      privileged: true
    volumeMounts:
      - name: docker-graph-storage
        mountPath: /var/lib/docker
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
"""
    }
  }
  stages {
    stage('Build With Dind')  {
      steps {
        container('dind') {
          sh '''
          dockerConfig=\${DOCKER_CONFIG:-~/.docker}
          [ -d \${dockerConfig} ] && echo "Docker directory Exists" || mkdir -p \${dockerConfig}
          echo '{"credsStore":"ecr-login"}' > \${dockerConfig}/config.json
          '''
          sh 'touch Dockerfile'
          sh 'echo "FROM centos:7" > Dockerfile'
          sh "cat Dockerfile"
          sh "docker build -t 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hello-dind:latest ."
          sh "docker push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hello-dind:latest"
        }
      }
    }
  }
}
▼

References