Jakarta EE Platform TCK CI is getting JNLP OOM (-Xmx256m JNLP Java memory setting may be too low).

Summary

We see java.lang.OutOfMemoryError: Java heap space in CI test job.

How can we increase the default memory settings for JNLP? Perhaps via:

https://github.com/eclipse-cbi/jiro/blob/master/instances/ee4j.jakartaee-tck/target/config.json (see -Xmx256m)

https://github.com/eclipse-cbi/jiro/blob/master/instances/ee4j.jakartaee-tck/target/jenkins/configuration.yml (see -Xmx256m

https://github.com/eclipse-cbi/jiro/blob/master/instances/ee4j.jakartaee-tck/target/k8s/configmap-jenkins-config.yml

Also note the jnlp container memory setting :

                containers:
                - name: jnlp
                  resources:
                    limits:
                      cpu: "4460m"
                      memory: "16384Mi"
                    requests:
                      cpu: "4160m"
                      memory: "16384Mi"

Please also note that our project repository has a Jenkinsfile but we aren't using that for the Jakarta EE 11 release which is why I opened this issue for changing the project defaults instead.

Steps to reproduce

(How one can reproduce the issue - this is very important)

Rebuild the https://ci.eclipse.org/jakartaee-tck/job/11/job/tck/job/jakarta-jdbc-tck-glassfish/59/ job and wait a few hours.

What is the current bug behavior?

We are likely seeing more test failures that cause a larger test result to be collected at the end of the test run and get the JNLP OOM because JNLP process max java memory is low.

(What actually happens) We see java.lang.OutOfMemoryError: Java heap space in CI test job

Should we try increasing the mentioned config files to use a higher -Xmx setting than 256m? Do the configuration files need to be regenerated or do you typically make manual changes to each one?

What is the expected correct behavior?

Should not see a java.lang.OutOfMemoryError: Java heap space JNLP error.

Relevant logs and/or screenshots

as described below from one of the CI test jobs:

ERROR: Step ‘Publish JUnit test result report’ aborted due to exception: 
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.40.60.50/10.40.60.50:32868
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1923)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
		at hudson.remoting.Channel.call(Channel.java:1112)
		at hudson.FilePath.act(FilePath.java:1207)
		at hudson.FilePath.act(FilePath.java:1196)
		at PluginClassLoader for junit//hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:146)
		at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:177)
		at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:282)
		at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:194)
		at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
		at hudson.model.Build$BuildExecution.post2(Build.java:179)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
		at hudson.model.Run.execute(Run.java:1856)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
		at hudson.model.ResourceController.execute(ResourceController.java:101)
		at hudson.model.Executor.run(Executor.java:446)
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3568)
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
	at java.base/java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1886)
	at java.base/java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1795)
	at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1192)
	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:350)
	at hudson.remoting.UserRequest._serialize(UserRequest.java:283)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:292)
	at hudson.remoting.UserRequest.perform(UserRequest.java:239)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:391)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:140)
	at java.base/java.lang.Thread.run(Thread.java:853)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from 10.40.60.50/10.40.60.50:32868 failed
	at hudson.remoting.Channel.call(Channel.java:1116)
	at hudson.FilePath.act(FilePath.java:1207)
	at hudson.FilePath.act(FilePath.java:1196)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:146)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:177)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:282)
	at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:194)
	at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
	at hudson.model.Build$BuildExecution.post2(Build.java:179)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
	at hudson.model.Run.execute(Run.java:1856)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:446)
[description-setter] Description set: full-21

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

We are trying to complete the Jakarta EE 11 release and resolve an OOM memory error.

added IT-prioritymedium IT-severitynormal service:jenkins statetodo team:releng labels

assigned to @pstankie

added statewip label and removed statetodo label

I increased the Xmx to 4G as test and rerunning https://ci.eclipse.org/jakartaee-tck/job/11/job/tck/job/jakarta-jdbc-tck-glassfish/60/console in order to catch statistcs.

Very nice looking results, thank you @pstankie! The increase of Xmx to 4G addressed the OOM!

I'm going to run a bigger test run.

It does seem like this change is still working.

Do the https://github.com/eclipse-cbi/jiro/blob/master/instances/ee4j.jakartaee-tck configuration files need to be updated also?

It does seem like this change is still working.

Did you mean this is still NOT working?

@fgurr it is working!

So for build https://ci.eclipse.org/jakartaee-tck/job/11/job/tck/job/jakarta-jdbc-tck-glassfish/62/ and dynamic agent basic-048jz memory usage for container jnlp looks like this:

Thanks @pstankie the ^ memory usage looks good!