Jenkins build fails after migrating to github

Summary

The jenkin builds of https://ci.eclipse.org/4diac/view/all/job/4diac%20IDE%20Develop/ are failing.

The only error message is:

Build step 'Invoke top-level Maven targets' marked build as failure Terminating xvnc.

Other builds had the same problem:

For example the 4diac IDE RC build.

https://ci.eclipse.org/4diac/view/all/job/4diac%20IDE%20R

here i added the debug flag -X . That magically solved the problem.

Unfortunately this didn't worked for the develop build.

The only difference i found out it that we have java 21 on develop.

Executing the build locally works perfectly fine.

Steps to reproduce

Trigger a build in https://ci.eclipse.org/4diac/view/all/job/4diac%20IDE%20Develop/

What is the current bug behavior?

Build not completed

What is the expected correct behavior?

Build runs successfull

Relevant logs and/or screenshots

https://ci.eclipse.org/4diac/view/all/job/4diac%20IDE%20Develop/

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

It is not able to provide updates and releases on all update sites

changed the description

added IT-prioritymedium IT-severitynormal service:GitHub service:jenkins statetodo team:releng labels

assigned to @fgurr

added statewip label and removed statetodo label

We are investigating.

I can't reproduce the issue at the moment, since the build is failing with a different error, see:
https://ci.eclipse.org/4diac/view/all/job/4diac%20IDE%20Develop/1283/console

@fgurr This happened in parallel because of an update of GEF Classic. This is fixed now. But the original problem remains.

ok i think this problem results from the fact that we got this commit from https://github.com/eclipse-4diac/4diac-ide/commit/1629376d765d09349ca79f5bc634a299ea5d8010 has been merged in the meantime to develop.

We will fix this first, and let you know.

It also seems that Poll scm is not working.

I can't confirm this yet.

In any case, I would recommend moving away from Freestyle jobs and polling SCM. Switching all jobs to Jenkins Multibranch pipeline and relying on webhooks to trigger builds is the better/best practice.

These are nightly builds. How to best trigger them else?

Nightly builds are outdated. A build should only be triggered when code changes.

We have way to many PRs per day. So I definitely don't want that. This has no value at all.

Interesting. 🤨

Even with a nightly build, I'd recommend switching to a pipeline build. I noticed that there is already a Jenkinsfile in the develop branch (https://github.com/eclipse-4diac/4diac-ide/blob/develop/Jenkinsfile) that looks like a good starting point.

SCM Polling is definitely not working for this Job: https://ci.eclipse.org/4diac/job/4diac%20IDE%20Milestone/

@fgurr any tip what we can do that we get our scm checking back. Previously we had it that the SCM was checked once a day if there are new commits on one of our three main work branches where done, and if there are, a build and deploy was started. This is very convenient for our beta testers and they would love to get that back.

With a test job, the SCM polling did definitely not work as expected.

While I still don't recommend polling or nightly builds, as a workaround I'd suggest using a periodical build "Build triggers -> Build periodically".

@fgurr I tried now different options. Neither SCM polling nor "Build triggers -> Build periodically" works. For me this seems to be a Jenkins server problem. Can you please look into that. We currently have no one that is capable of setting up Jenkins Multibranch pipeline with webhooks and our previous setup did exactly what we need for the last 6 years or more.

https://ci.eclipse.org/4diac/view/all/job/4diac%20IDE%20Develop/ runs successfully with semeru-jdk21-latest. This is a strong hint that the issue is related to memory issues. The obvious and common clue of a "Killed" line in the console log is weirdly missing.

Interesting. I already doubled the Xmx setting for maven.

I see that the latest builds are green. Can we close the issue? or is there anything more to do?

I see that periodic builds are working. May I try to to also try a SCM poll setup?

Sure, give it a try!

May I suggest the same approach as recommended by Fred? Multibranch pipeline and webhook. Looking at the job configuration, I don't think this is a huge rework with so many benefits.

This is how CI works nowadays!

As said no one with expertise and time and it would result in much more builds then with our setup. So I still think our setup is much better for EF.

Ok it is up to you.

One last thing, 4diac project has 1 resource pack, which means 4CPU and 8GiB. And looking at our dashboard for the last 7 days, this is underuse compared to what we offer.

@heurtemattes no time triggered builds are working neither poll SCM nor build periodical is working.

For sure the resource pack is underused. Our builds are not working!

And even if they would it currently offers more then what we need.

no time triggered builds are working neither poll SCM nor build periodical is working.

Thanks for testing this, I'll investigate and do more tests next Monday.

And even if they would it currently offers more then what we need.

Interesting, your feedback on this ticket would be valuable. #4759

no time triggered builds are working neither poll SCM nor build periodical is working.

Thanks for testing this, I'll investigate and do more tests next Monday.

Thanks!

And even if they would it currently offers more then what we need.

Interesting, your feedback on this ticket would be valuable. #4759

I fear we would be a bad testimonial. One of the reasons we don't need more is that we rely on github workflows for our PR integration and test builds. JIPP we now only use for our nightly, milestone, and RC builds, so that we get Eclipse signing and deployment to download.eclipse.org.

Maybe one other reason is that we still have way to little tests.

On the other hand we also have a rather simple build, with mostly only dependencies to SimRel components. Not sure if this has an impact here.

I found this message in the admin panel:

The following triggers took a long time to process. This is usually caused by networking issues or high load on the Jenkins controller. As a result, other triggers could be delayed and scheduled tasks may not have started at the expected time.

I wonder if there is a conflict with another plugin. Is the migration from gerrit to github complete? we could try to disable all gerrit jobs, and remove the gerrit plugin.

At the same time, I have set log FINEST for the scheduler. We'll see if we can get relevant data.

There is also this SEVERE message in the log. This is another lead.

2024-07-01 12:10:38.829+0000 [id=45] INFO o.j.p.ghprb.GhprbPullRequest#<init>: Author of #237 pazi146 on eclipse-4diac/4diac-ide not in whitelist!
2024-07-01 12:10:39.095+0000 [id=45] SEVERE o.j.p.ghprb.GhprbRepository#addComment: Could not add comment to pull request #237: 'Can one of the admins verify this patch?'
org.kohsuke.github.HttpException: {"message":"Requires authentication","documentation_url":"https://docs.github.com/rest/issues/comments#create-an-issue-comment","status":"401"}
at org.kohsuke.github.GitHubConnectorResponseErrorHandler$1.onError(GitHubConnectorResponseErrorHandler.java:62)
at org.kohsuke.github.GitHubClient.detectKnownErrors(GitHubClient.java:504)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:464)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:427)
at org.kohsuke.github.Requester.fetch(Requester.java:85)
``

Yes our migration to github is complete. So we would be totally fine to remove the gerrit plugin from our Jenkins jobs. But the jobs in question where never gerrit jobs. Not sure if that matters.

After reading the log file content I got suspicious. I now either disabled or deleted all outdated and not used Jenkins jobs. With your feedback I somehow had the feeling that they are maybe interfering. Let's see.

I now either disabled or deleted all outdated and not used Jenkins jobs.

It looks like it freed the queue. Jobs are now able to run periodically again.

I have reconfigured the job with the previous schedule crontime: H 20 * * *, for my test it ran every 5min.

The Gerrit-trigger plugin has been removed.

Let's wait for the next periodic run, I am quite confident

From what I see I'm also pretty confident. I would even go that far that poll scm will come back.

Thx a lot for digging into that. I would never have seen that. BTW is there a possibility that we can also see these logs?

From what I see I'm also pretty confident. I would even go that far that poll scm will come back.

BTW is there a possibility that we can also see these logs?

Unfortunately, these logs are from the Jenkins controller, accessible via the cluster or the admin panel.

We're really thinking about a way to provide more metrics and logs to communities.

But in any case, do not hesitate to ask us, we can provide them to you.

I feared so. Then I can only repeat my thx for your support!

The poll configuration has worked for the latest builds.

I'm closing the ticket, If you experience further issues, please don't hesitate to reopen it.

you where faster then me. I was about to write that. Thx again.

removed statewip label

assigned to @heurtemattes

changed due date to July 02, 2024

closed