Issue
-
I want to configure the artifact S3 manager plugin
-
I would like to store the artifacts into AWS S3
Environment
Resolution
As stated in CloudBees Core Performance Best Practices: Do not use the standard Archive the Artifacts post-build step for large artifacts (> 100KB), they should be sent to your favorite Artifact Repository Manager e.g. Artifactory, Nexus, S3, etc.).
Following is a step-by-step guide to store the Jenkins artifacts into AWS S3 by using Artifact Manager on S3.
Prerequisites in AWS
A. A user (e.g. s3-artifacts
) with the right policies for S3. For this example we are using arn:aws:iam::aws:policy/AmazonS3FullAccess
but you can be more restrictive as explained in the plugin page.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
B. An S3 bucket (e.g. artifact-bucket-example
). Ideally, a separate folder to store your artifacts (e.g. acme-artifacts
).
Steps in CloudBees Core
1. Create an AWS Crendential
type in Jenkins using id
of the user (s3-artifacts
) plus its access_key_id
and its aws_secret_access_key
.
2. Configure the controller to store all the artifacts in S3 via Manage Jenkins > Configure System > Artifact Management for Builds > Cloud Artifact Storage > Cloud Provider > Amazon S3 and Save
the configuration.
3. Configure the plugin via Manage Jenkins > AWS
-
S3 Bucket Name:
artifact-bucket-example
-
Base Prefix:
acme-artifacts/
-
Amazon Credentials (step 1)
Then, Validate S3 Bucket configuration
. Expected value is success
.
4. Run a demo Pipeline as the following exmaple:
pipeline {
agent {
kubernetes {
containerTemplate {
name 'agent'
image 'jenkins/agent:alpine'
ttyEnabled true
}
}
}
stages {
stage('archiving artifacts into AWS s3') {
steps {
script {
sh "dd if=/dev/urandom of=artifact.txt bs=5MB count=1"
}
archiveArtifacts artifacts: "*.*", fingerprint: true
}
}
}
}
Expected console output:
[Pipeline] { (archiving artifacts into AWS s3)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ dd 'if=/dev/urandom' 'of=artifact.txt' 'bs=5MB' 'count=1'
1+0 records in
1+0 records out
[Pipeline] }
[Pipeline] // script
[Pipeline] archiveArtifacts
Archiving artifacts
Uploaded 1 artifact(s) to https://artifact-bucket-example.s3.amazonaws.com/acme-artifacts/pipeline-demo/1/artifacts/
Recording fingerprints
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
Troubleshooting
1. Ensure that Amazon S3 has been defined correctly, Manage Jenkins > Configure System > Artifact Management for Builds > Cloud Artifact Storage > Cloud Provider > Amazon S3
2. Create a custom logger for Artifact Manager on S3
(important: delete this log once it is not needed)
-
io.jenkins.plugins.artifact_manager_jclouds
- ALL -
org.jclouds.rest.internal.InvokeHttpMethod
- FINE
3. Use the AWS CLI locally
-
Create the file
$> dd if=/dev/urandom of=artifact.txt bs=5MB count=1
1+0 records in
1+0 records out
5000000 bytes (5.0 MB, 4.8 MiB) copied, 0.0481165 s, 104 MB/s
-
Include
~/.aws/config
[profile s3-artifacts]
aws_access_key_id = AKI*************
aws_secret_access_key = Uce**************************
-
Run the AWS CLI
aws --profile cloudbees-support-s3 s3 cp artifact.txt s3://artifact-bucket-example/my-artifacts/artifact-yeah.txt
upload: ./artifact.txt to s3://artifacts-bucket-carlosrodlop/my-artifacts/artifact-yeah.txt