No space left on device during Docker build for Eclipse SDK

Summary

Docker container build broken which is required for Eclipse SDk builds

See: https://github.com/eclipse-platform/eclipse.platform.releng/issues/210

In the docker logs, lines like this are found (see https://ci.eclipse.org/releng/job/Build-Docker-images/498/ ):

error: unpacking of archive failed on file /usr/include/gtk-3.0/gdk/x11/gdkx11devicemanager.h: cpio: rename failed - No space left on device

The produced Eclipse build is therefore broken.

Please clean up enough space on whichever machines these jobs are running.

Steps to reproduce

Not known if its reproducible and whether it will occur on every Eclipse integration build from now on.

What is the current bug behavior?

The Eclipse build ran out of space

What is the expected correct behavior?

There is enough space to build Eclipse.

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

Eclipse build is broken.

changed title from No space left on device during Eclipse integration build to No space left on device during Docker build for Eclipse SDK

changed the description

@fgurr @mbarbero : I assume this is a trivial task to fix, would it be possible to do this today? SDK builds on Linux are all broken because of broken docker container builds.

added IT-prioritymedium IT-severitynormal ~55077 ~24521 team:releng labels

assigned to @fgurr

There are currently 9.2GB left on "/".

It seems like the build is exhausting those 9.2GB during the docker builds. I would recommend splitting up the build to fix this.

I would recommend splitting up the build to fix this.

Could you please elaborate how splitting the build can be done and why it could help?

Isn't same work that need to be done? May be I simply lack knowledge about docker builds. Or does the build run some tasks in parallel somehow?

AFAICT the job builds 10 different docker images in one run. If it was split into two jobs that build only 5 docker images, the required disk space (during a build) could be reduced. The job is already cleaning up before and after the docker builds.

What about "cleaning up" between image builds? Or do we need multiple Jenkins jobs because that cleanup can't be done in the middle of such job?

Sure. That could work as well.

Thanks Fred. Last question : where is the job source code ?

=> https://ci.eclipse.org/releng/job/Build-Docker-images/configure

OMG. @slakkimsetti, @akurtakov : do we really need so many images?

I see centos7, 8, 9, few ubuntu etc? Is there any documentation which image is configured for which reason? Before splitting all that, shouldn't we simply throw away unused images?

I'm all for reducing images. I've tried to replace centos7 and the only instance left I am aware of is https://github.com/eclipse-platform/eclipse.platform.text/blob/master/Jenkinsfile#L8 as platform.text tests were not working on centos7 last that I tried. Ubuntu and Suse images are used by https://ci.eclipse.org/releng/job/Start-smoke-tests/ but due to exhausting resources only latest Ubuntu is used. I'll get rid of Ubuntu 18 and 20 images now and if someone migrates platform.text verification build to Centos8 we can get rid of Centos7 too. It's a whole new story if someone would volunteer to move all verification builds to Cent9 so Cent8 could be dropped too and we have only latest version of all 3 major linux distros.

No more Ubuntu 18/20 builds https://ci.eclipse.org/releng/job/Build-Docker-images/jobConfigHistory/showDiffFiles?timestamp1=2022-06-14_02-13-47&timestamp2=2023-03-17_02-12-25 . Build running as https://ci.eclipse.org/releng/job/Build-Docker-images/501/ , if someone can verify that everything built fine that would be nice.

Platform text effort to drop centos7 is at https://github.com/eclipse-platform/eclipse.platform.text/pull/162 .

Alex, do you want first to merge https://github.com/eclipse-platform/eclipse.platform.text/pull/162 or can we restart Linux jobs now?

https://ci.eclipse.org/releng/job/Build-Docker-images/501/consoleFull seem to be successful.

Platform.text breaks the tests so I'm not merging it. I've restarted only the Java 17 tests https://ci.eclipse.org/releng/job/AutomatedTests/job/ep428I-unit-cen64-gtk3-java17/16/ so we don't put too much stress on the build machines. If this succeeds, Java 19 tests would have to be restarted too.

Job failed immediately with

‘centos-unitpod17-svbrl-3qfnl’ is offline
releng/centos-unitpod17-svbrl-3qfnl Container jnlp was terminated (Exit Code: 255, Reason: Error)

- custom -- running
-----Logs-------------


- jnlp -- terminated (255)
-----Logs-------------
Mar 17, 2023 6:51:25 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3044.vb_940a_a_e4f72e
Mar 17, 2023 6:51:25 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Mar 17, 2023 6:51:25 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Mar 17, 2023 6:51:25 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-ui.releng.svc.cluster.local/releng/]
Mar 17, 2023 6:51:45 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Failed to connect to http://jenkins-ui.releng.svc.cluster.local/releng/tcpSlaveAgentListener/: jenkins-ui.releng.svc.cluster.local
java.io.IOException: Failed to connect to http://jenkins-ui.releng.svc.cluster.local/releng/tcpSlaveAgentListener/: jenkins-ui.releng.svc.cluster.local
	at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:214)
	at hudson.remoting.Engine.innerRun(Engine.java:744)
	at hudson.remoting.Engine.run(Engine.java:543)
Caused by: java.net.UnknownHostException: jenkins-ui.releng.svc.cluster.local
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:275)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:374)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:395)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:211)
	... 2 more

This issue of randomly failing to connect is with us for months. @fgurr IIRC you mentioned some updates are supposed to fix it, right? Are they performed?

Alex, this seem to be different issue now, it looks like job wants a node that is either not running or does not exist (anymore?):

‘centos-unitpod17-svbrl-3qfnl’ is offline java.net.UnknownHostException: jenkins-ui.releng.svc.cluster.local

Hmm, I see next job found some node: https://ci.eclipse.org/releng/job/AutomatedTests/job/ep428I-unit-cen64-gtk3-java17/17/consoleFull

This issue of randomly failing to connect is with us for months. @fgurr IIRC you mentioned some updates are supposed to fix it, right? Are they performed?

Jenkins and Kubernetes Plugin were updated on Releng Jenkins instance, but it seems like it did not help much.

We have a cleaning script that cleans all the workspaces https://ci.eclipse.org/releng/job/Cleanup/job/cleanup-performance-tests-machine/ There was a problem in this script after we restructured and moved all the jenkins jobs as code. Right now I fixed the job.

And the docker build should work now. I cleaned about 11 GB now.

Thanks Sravan.

So I just tried running the build on CentOS 9 in https://ci.eclipse.org/platform/job/eclipse.platform.text/job/PR-164/1/console but it fails with "There are no nodes with the label ‘centos-9’" . There is something I'm missing here.

@slakkimsetti So docker images build on releng JIPP are not generally usable like I tried but only for smoke tests? If that's the case centos7 is smth I should drop too, right?

@fgurr Is there supposed to be generally usable centos-9 label to run builds on ?

@slakkimsetti So docker images build on releng JIPP are not generally usable like I tried but only for smoke tests? If that's the case centos7 is smth I should drop too, right?

I used these images in testing only. Any of these images can be used in test jobs. Looking at https://github.com/eclipse-platform/eclipse.platform.text/pull/164/files I see the problem. There are two objects when we work with jenkins on OKD.

1. Pod-template (we have multiple pod templates but all are based on either centos-7 or centos-8)
2. Images.

The change you made in https://github.com/eclipse-platform/eclipse.platform.text/pull/164/files requires a pod -template with name centos-9 which we don't have. That is the problem you are seeing.

In case of test jobs we define pod definition in the form of yaml. So we can use any image we like.

So I can stop building centos7 docker images as the one used by platform.text still is different from the one we build?

yes you can stop building.

Console log shows log from all the 10 docker build commands. It will be difficult to identify the exact problem. Individual logs are archived and available as build artifacts. If there is any error please go through them. they are separated per docker image.

@fgurr Is there supposed to be generally usable centos-9 label to run builds on ?

No, and there are no plans to make a CentOS 9 image available as a default pod template. We are aiming to switch to Ubuntu-based images (see #1623), it just got buried in the backlog.

Ok, the docker images are rebuilt now successfully, we have first "green" build on Linux since days, I believe this issue can be closed?

+1 Now that we build 2 less Ubuntu and 1 less CentOS images things should be far better.

Thx for cleaning up, guys.

closed