EC2-Plugin Out of Memory issue when launching agents in AWS

Issue

We have EC2-plugin that is launching m3.large dynamic agents in AWS. However, we are running into memory issues consistently below. Also, When we manually attach the agent to CloudBees Enterprise server using agent.jnlp the jobs run perfectly fine.

The jobs are failing because can not allocate memory at agents:

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 855376 bytes for Chunk::new
An error report file with more information is saved as:

Error occurred during initialization of VM
Could not reserve enough space for object heap

Jenkins log shows errors like:

2015-12-08 15:28:33.726+0000 [id=337861]	SEVERE	h.p.ec2.win.winrm.WinRMClient#sendRequest: I/O Exception in HTTP POST
org.apache.http.conn.ConnectTimeoutException: Connect to 10.71.6.223:5985 timed out
	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:119)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:231)
	at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:202)
	at hudson.plugins.ec2.win.winrm.WinRMClient.openShell(WinRMClient.java:85)
	at hudson.plugins.ec2.win.winrm.WinRM.ping(WinRM.java:29)
	at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:117)
	at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:123)
	at hudson.plugins.ec2.win.EC2WindowsLauncher.launch(EC2WindowsLauncher.java:31)
	at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:107)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:243)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Environment

CloudBees Jenkins Enterprise

EC2 Plugin

AWS

Resolution

WinRM limits the amount of RAM available to the "remote shell"

https://learn.microsoft.com/en-us/windows/win32/winrm/quotas?redirectedfrom=MSDN

the EC2 plugin launches windows agents via WinRM.

These limits will need configuring if you want more than 1GB of RAM to be used by Jenkins and all processes it starts.

e.g. as an Administrator using an elevated command prompt execute winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="10240"}' Note you may also require to set the number of processes as well depending on the number of executors and types of jobs run.