Setting readYaml max aliases for collections system properties via Java arguments does not work

2 minute readKnowledge base

Issue

If I start up a controller, with the following plugin installed:

and use the readYaml step with a yaml containing 51 aliases/anchors:

pipeline {
    agent any

    stages {
        stage('Hello') {
            steps {
                readYaml file: 'moreThan50anchors.yml'
            }
        }
    }
}
▼

I get the expected error, due to SnakeYAML’s default configuration of 50, and my yaml containing 51 aliases:

org.yaml.snakeyaml.error.YAMLException: Number of aliases for non-scalar nodes exceeds the specified max=50
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:193)
    ...
	at org.yaml.snakeyaml.Yaml$1.next(Yaml.java:499)
	at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep$Execution.doRun(ReadYamlStep.java:203)
	...

Thanks to the fix from JENKINS-68830 I can then go to Manage Jenkins → Script console and run:

org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.DEFAULT_MAX_ALIASES_FOR_COLLECTIONS=51
org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.MAX_MAX_ALIASES_FOR_COLLECTIONS=51
▼

And run the same pipeline again, it is successful.

Unexpected behaviour

Instead of using the script console, if I add these as startup arguments:

java -Dorg.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.DEFAULT_MAX_ALIASES_FOR_COLLECTIONS=51 -Dorg.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.MAX_MAX_ALIASES_FOR_COLLECTIONS=51 -jar cloudbees-core-cm.war

Then I run my same pipeline (without running anything in the script console first), I get the error in the Pipeline log:

java.lang.IllegalArgumentException: 51 > 0. Reduce the required DEFAULT_MAX_ALIASES_FOR_COLLECTIONS or convince your administrator to increase the max allowed value with the system property "null"
	at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.setDefaultMaxAliasesForCollections(ReadYamlStep.java:105)
	at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.<clinit>(ReadYamlStep.java:68)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    ...

This exception is coming from: https://github.com/jenkinsci/pipeline-utility-steps-plugin/blob/0326b3cb497dee95348471fca1c7c62dbb6975fb/src/main/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/ReadYamlStep.java#L105

Then every subsequent pipeline run after that one encounters the error:

java.lang.NoClassDefFoundError: Could not initialize class org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	...

Environment

Resolution

This issue was fixed in pipeline-utility-steps version 2.15.1

This bug was tracked under https://issues.jenkins.io/browse/JENKINS-70557.

Ideally, we should not be parsing such a large yaml file using the readYaml step, This command loads the data into the controller’s memory, and uses resources on the controller, which at scale (if the yaml is very large, or it runs very often from multiple pipelines, for example using a Pipeline shared library) can cause performance issues.

We should change the Pipeline code to use sh 'yq …​' to use a shell step on an agent to run yq (as also recommended in the plugin documentation).

Workaround

Instead of setting the system properties via startup arguments, set them via the script console.

You can also set this as a groovy post initialization script, by creating a file JENKINS_HOME/init.groovy.d/readYamlAliases.groovy with the content:

org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.DEFAULT_MAX_ALIASES_FOR_COLLECTIONS=51
org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.MAX_MAX_ALIASES_FOR_COLLECTIONS=51
▼

Adjusting 51 to whatever higher limit you require, based on the number of aliases you have.

Tested product/plugin versions