Integration with AWS S3 as bucket for your Jenkins artifacts

Article ID:360041265712
2 minute readKnowledge base

Issue

  • I want to configure the artifact S3 manager plugin

  • I would like to store the artifacts into AWS S3

Environment

Resolution

As stated in CloudBees Core Performance Best Practices: Do not use the standard Archive the Artifacts post-build step for large artifacts (> 100KB), they should be sent to your favorite Artifact Repository Manager e.g. Artifactory, Nexus, S3, etc.).

Following is a step-by-step guide to store the Jenkins artifacts into AWS S3 by using Artifact Manager on S3.

Prerequisites in AWS

A. A user (e.g. s3-artifacts) with the right policies for S3. For this example we are using arn:aws:iam::aws:policy/AmazonS3FullAccess but you can be more restrictive as explained in the plugin page.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}
▼

B. An S3 bucket (e.g. artifact-bucket-example). Ideally, a separate folder to store your artifacts (e.g. acme-artifacts).

Steps in CloudBees Core

1. Create an AWS Crendential type in Jenkins using id of the user (s3-artifacts) plus its access_key_id and its aws_secret_access_key.

2. Configure the controller to store all the artifacts in S3 via Manage Jenkins > Configure System > Artifact Management for Builds > Cloud Artifact Storage > Cloud Provider > Amazon S3 and Save the configuration.

3. Configure the plugin via Manage Jenkins > AWS

  • S3 Bucket Name: artifact-bucket-example

  • Base Prefix: acme-artifacts/

  • Amazon Credentials (step 1)

Then, Validate S3 Bucket configuration. Expected value is success.

4. Run a demo Pipeline as the following exmaple:

pipeline {
    agent {
        kubernetes {
          containerTemplate {
            name 'agent'
            image 'jenkins/agent:alpine'
            ttyEnabled true
            }
        }
    }
    stages {
        stage('archiving artifacts into AWS s3') {
            steps {
                script {
                    sh "dd if=/dev/urandom of=artifact.txt bs=5MB count=1"
                }
                archiveArtifacts artifacts: "*.*", fingerprint: true
            }
        }
    }
}
▼

Expected console output:

[Pipeline] { (archiving artifacts into AWS s3)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ dd 'if=/dev/urandom' 'of=artifact.txt' 'bs=5MB' 'count=1'
1+0 records in
1+0 records out
[Pipeline] }
[Pipeline] // script
[Pipeline] archiveArtifacts
Archiving artifacts
Uploaded 1 artifact(s) to https://artifact-bucket-example.s3.amazonaws.com/acme-artifacts/pipeline-demo/1/artifacts/
Recording fingerprints
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
▼

Troubleshooting

1. Ensure that Amazon S3 has been defined correctly, Manage Jenkins > Configure System > Artifact Management for Builds > Cloud Artifact Storage > Cloud Provider > Amazon S3

2. Create a custom logger for Artifact Manager on S3 (important: delete this log once it is not needed)

  • io.jenkins.plugins.artifact_manager_jclouds - ALL

  • org.jclouds.rest.internal.InvokeHttpMethod - FINE

3. Use the AWS CLI locally

  • Create the file

$> dd if=/dev/urandom of=artifact.txt bs=5MB count=1
1+0 records in
1+0 records out
5000000 bytes (5.0 MB, 4.8 MiB) copied, 0.0481165 s, 104 MB/s
▼

  • Include ~/.aws/config

[profile s3-artifacts]
aws_access_key_id     = AKI*************
aws_secret_access_key = Uce**************************
▼
aws --profile cloudbees-support-s3 s3 cp artifact.txt s3://artifact-bucket-example/my-artifacts/artifact-yeah.txt
upload: ./artifact.txt to s3://artifacts-bucket-carlosrodlop/my-artifacts/artifact-yeah.txt
▼

Tested products/plugins version

The latest update of this article was tested with:

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.