Explanation
AWS IAM roles for service accounts (IRSA) allows to bind a Kubernetes ServiceAccount to IAM Roles, that allows fine-grained authorization within AWS. When building and pushing docker images from Jenkins agent to a private registries like AWS ECR, specific kubernetes ServiceAccounts can be used to provide the necessary permissions in AWS. But attaching the ServiceAccount to agent pods is not all that is needed. Further configuration is required to make sure that the tools used (Kaniko, docker, …) are able to infer the AWS Identify from the environment.
Resolution
The solution is applicable to DinD
(but not DooD
- when mounting the docker socket) or a tool like Kaniko as it relies on the AWS ECR Credentials Helper.
Pre-Requisites
AWS / Kubernetes
As explained in AWS IAM roles for service accounts (IRSA):
-
There is an IAM Role that provides sufficient permission to pull / push to AWS ECR
-
There is an ServiceAccount created in the namespace where the Agent Pods are being deployed that are associated with the IAM Role
If using Kaniko
-
Kaniko version 0.16.0 or later must be used. Note: The Kaniko image at the time of writing already contains the docker credentials helper for AWS ECR. So nothing else is needed.
If using DinD
-
The AWS ECR Credentials Helper version 0.4.0 of later is installed and available in the
dind
container. The following is an example of aDockerfile
that creates adind
image with the AWS ECR Credentials Helper added to thePATH
:# Stage 0: Get credential helpers FROM golang:1.15 RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login RUN make -C /go/src/github.com/awslabs/amazon-ecr-credential-helper linux-amd64 FROM docker:19.03.11-dind COPY --from=0 /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/linux-amd64/docker-credential-ecr-login /usr/local/bin/docker-credential-ecr-login
Steps
In short, the following steps are required:
-
Attach the ServiceAccount that is associated to the IAM Role to the Agent Pod
.spec.serviceAccountName
-
Inject the environment variables
AWS_EC2_METADATA_DISABLED=true
andAWS_SDK_LOAD_CONFIG=true
to the "tool" container in the Pod. This is specific to the behavior of the AWS ECR Credentials Helper. -
Add the ECR credentials helper configuration (
'{"credsStore":"ecr-login"}'
) in the docker config file (by default in the container userHOME
directory ~/.docker/config.json`)
Examples
Kaniko
Here is an overall solution based on the documentation at Using Kaniko with CloudBees Core:
pipeline { agent { kubernetes { yaml """ kind: Pod metadata: name: kaniko spec: serviceAccountName: aws-iam-ecr containers: - name: kaniko image: gcr.io/kaniko-project/executor:debug imagePullPolicy: Always command: - sleep args: - 9999999 env: - name: AWS_EC2_METADATA_DISABLED value: true - name: AWS_SDK_LOAD_CONFIG value: true """ } } stages { stage('Build with Kaniko') { steps { container(name: 'kaniko', shell: '/busybox/sh') { sh '''#!/busybox/sh dockerConfig=\${DOCKER_CONFIG:-/kaniko/.docker} [ -d \${dockerConfig} ] && echo "Docker directory Exists" || mkdir -p \${dockerConfig} echo '{"credsStore":"ecr-login"}' > \${dockerConfig}/config.json ''' sh '''#!/busybox/sh echo "FROM jenkins/inbound-agent:latest" > Dockerfile /kaniko/executor --context `pwd` --destination 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hello-kaniko:latest ''' } } } } }
DinD
Here is an overall solution based on the documentation at How to build my own docker images in CloudBees CI (CloudBees Core) on Modern Cloud Platforms:
pipeline { agent { kubernetes { yaml """ apiVersion: v1 kind: Pod spec: serviceAccountName: aws-iam-ecr containers: - name: dind # `dind` image with ECR helper image: my-dind-with-ecr:19.03.11 imagePullPolicy: Always env: - name: AWS_EC2_METADATA_DISABLED value: true - name: AWS_SDK_LOAD_CONFIG value: true tty: true securityContext: privileged: true volumeMounts: - name: docker-graph-storage mountPath: /var/lib/docker volumes: - name: docker-graph-storage emptyDir: {} """ } } stages { stage('Build With Dind') { steps { container('dind') { sh ''' dockerConfig=\${DOCKER_CONFIG:-~/.docker} [ -d \${dockerConfig} ] && echo "Docker directory Exists" || mkdir -p \${dockerConfig} echo '{"credsStore":"ecr-login"}' > \${dockerConfig}/config.json ''' sh 'touch Dockerfile' sh 'echo "FROM centos:7" > Dockerfile' sh "cat Dockerfile" sh "docker build -t 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hello-dind:latest ." sh "docker push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hello-dind:latest" } } } } }