Eclipse Jenkins builds do not run

Summary

The Eclipse Jenkins for Eclipse Kura seems stuck. We have more than 20 jobs in the waiting list to be built. The usual 2 executors that should pick the jobs in the queue list are not available and even not displayed

Steps to reproduce

Access the Eclipse Jenkins for Eclipse Kura https://ci.eclipse.org/kura/ on the left part you'll see the queue for build filled with jobs and in the lower left part no running executors

What is the current bug behavior?

No build is performed. The project has multiple pull request waiting

What is the expected correct behavior?

Build should be pipelined and executed

Relevant logs and/or screenshots

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

The Eclipse Kura project is currently blocked as no pull request can be built. Pull requests, as a result cannot be merged.

added IT-prioritymedium IT-severitynormal ~24521 team:releng labels

assigned to @fgurr

JIPP has been restarted. Builds are running again.

closed

I'm sorry, I have to reopen it again. Now the build works but we are running on only one executor instead of the usual 2 and it is making the whole process extremely slow and tedious. Can you have please a check to see if an action needs to be taken?

Thanks.

reopened

After another restart two builds can run in parallel again.

closed

Thanks. Unfortunately it has worked for few hours but now we are back to a single executor

I've updated the Jenkins instance to a newer LTS version and updated some plugins. Let's see if that helps.

Hello, today the Kura Ci is again without running executors.

reopened

JIPP has been restarted. I've tweaked some settings to see if it makes a difference.

closed

Hello, sorry to bother. The executors are unfortunately down also today.

reopened

JIPP has been restarted.

closed

Hello @fgurr

unfortunately the executors are down again today.

On a (probably) related note: we noticed that there seems to be a periodic build active for our project. Every day it builds all the open PRs for our repo which is kinda wasteful and (looks like) increases the chances of Jenkins tripping. Can we do something for that?

reopened

I've restarted the Kura Jenkins instance and builds are running again.

On a (probably) related note: we noticed that there seems to be a periodic build active for our project. Every day it builds all the open PRs for our repo which is kinda wasteful and (looks like) increases the chances of Jenkins tripping. Can we do something for that?

In your multibranch configuration (https://ci.eclipse.org/kura/job/multibranch/configure), the option "Periodically if not otherwise run -> Interval -> 1 day" is set. So every day, all Branches and PRs are indexed. Builds should only be triggered if something has changed.

Since there are 30 open PRs it's hard to tell if builds are triggered more often than necessary. If you can pinpoint such a case, please let us know.

closed

I've restarted the Kura Jenkins instance and builds are running again.

Thank you for the quick feedback

Since there are 30 open PRs it's hard to tell if builds are triggered more often than necessary. If you can pinpoint such a case, please let us know.

This is a screenshot of the queue taken yesterday

PR 2534 is 4 years old at this point and nothing changed for that but it's still in the queue. Apart from this, the behaviour is what we expect from our settings.

marked #2826 (closed) as a duplicate of this issue

marked this issue as related to #2826 (closed)

Hi @fgurr

Since this issue is reappearing regularly on our CI (it is a weekly recurrence lately) and it is fairly painful to deal with, do you have any suggestion on how to avoid this? Is there anything we can do?

There are several issues:

• I don't know why old PRs like https://github.com/eclipse/kura/pull/2534 keep getting scheduled. The "Scan repository log" somehow detects a change:

    Checking pull request #2534
      ‘Jenkinsfile’ found
    Met criteria
Changes detected: PR-2534 (40b6d1f86e0c32d2d7a017b3ede22520e92e1727+4ccc70c140d39fa27cb18035af3feb452b8d7631 (56c383c8b59ce9b22baf35931e7cc2f3a07b922a) → 
40b6d1f86e0c32d2d7a017b3ede22520e92e1727+d52adb481d8342375361e0d38b6d413622a26579 (f8d6d7aca4eda6da8f7abdbc0c5b9dedcf71508a))
Scheduled build for branch: PR-2534

• There is a known issue with the Jenkins Kubernetes plugin that can lead to stuck builds, like you regularly experience at the moment. Usually this occurs only occasionally. Your CI instance seems to be affected a lot more often, though.
• The issue described in #2816 (closed) lead to failed builds.
  • => this is hopefully fixed now

My general recommendation for putting less pressure on the Jenkins instance is to reduce the number of open PRs. Especially old or outdated PRs.

reopened

Hi @fgurr

thank you for the insight. We got rid of older PRs as you suggested. Hopefully this will improve our experience.

@fgurr sorry for bothering you again but it seems that the Jenkins instance is stuck again today:

Jenkins instance has been restarted.

closed

@fgurr Hello there!

I think you already know what I'm about to say xD

Jenkins is stuck again. No excecutors and build are piling up...

Jenkins instance has been restarted.

Thanks. Everything seems to be working now.

@fgurr Hello there!

Well... it's been a while since last time isn't it?

Currently seeing no executor:

Jenkins has been restarted

Thanks. Everything seems to be working now

@fgurr Hello!

We're experiencing the usual issue with the excecutors... but hey, it seems that the situation is improving. At least I'm not pestering you every other week

As you might notice we applied your suggestion and got rid of all the old open PRs reducing the load on the Jenkins instance.

We're experiencing the usual issue with the excecutors... but hey, it seems that the situation is improving. At least I'm not pestering you every other week

I was starting to miss our conversations.

As you might notice we applied your suggestion and got rid of all the old open PRs reducing the load on the Jenkins instance.

The Kura Jenkins instance has been restarted. Build is running again.

Thanks for the quick feedback. Everything seems to be working

Hello there!

I was starting to miss our conversations.

@fgurr be careful what you wish for... Jenkins might be listening

... so yeah... we lost our executors again...

Edit: I just found out about the ongoing maintenance. Could be related to that...

Jenkins instance has been restarted.

Thanks. Everything is running now

reopened

closed

Hello there @fgurr !

We're currently experiencing the usual issue: no executors and builds are piling up.

Not sure if someone restarted the instance in the meantime, but I see builds running right now.

I have restarted jenkins 1 hour ago. everything looks good now.

Yup! Builds seems to be building alright. Thanks for your support

mentioned in issue #4380

Hello there @fgurr!

... yeah... you know the drill

I have cleaned the build queue and restarted the instance. Let me know if there is still a problem.

Thanks for the quick turnaround . Everything seems to be running fine now

Hello, I think we may have the same problem again today. Can you guys please have a check? Thanks

@mmaiero I have done some cleaning and it looks better now! builds restart. I'll keep an eye on it.

Hello there!

Looks like we have the usual problem:

Hi @fgurr @heurtemattes, any news on this? Currently we are stuck with the PRs due to this problem with the CI.

Best regards

@mmaiero As a quick workaround I did the usual thing - restarted Jenkins. I don't have long term solution but will also add this problem to my watchlist.

Thanks. Appreciated

Hello, it seems that the Eclipse Kura Jenkins is again stuck with no available executors. Could you please have a look? Thanks

@fgurr @pstankie @heurtemattes Hello! Any update on this? Our CI is still broken and PR are blocked.

reopened

added statewip label

@mmaiero I restarted master and there's job running: https://ci.eclipse.org/kura/job/multibranch/job/PR-5424/

Build queue is large though. I haven't cleaned it up.

Hi, thanks. It seems that we have again problems. There were PRs building, then suddenly all failed. Now it seems that only one executor is running

I restarted master again.

Hello @pstankie the problema is happening again on the instance for Kura

Hello, the Jenkins instance has become irresponsive and triggering a new build fails. Could you please have a look?

Thanks

@mmaiero I think all is good. There's at least 1 job running: https://ci.eclipse.org/kura/job/multibranch/indexing/events

And also there are 2 freshly started jenkins agents:

I'll keep an eye on it anyway.

Currently there's 1 job running: https://ci.eclipse.org/kura/job/multibranch/job/release-5.6.0/25/consoleFull

and one agent deployed.

The page https://ci.eclipse.org/kura/view/all/builds

Shows correct information.

Please reopen if more help is needed.

Hello, it seems that it got stuck again:

Build are running again.

Meantime, would it be possible to reduce build history size for all jobs to something like 5? we would like to transfer this instance to the new backend storage.

Once the transfer is complete we can restore those values.

Thanks for your understanding.

unfortunately it seems not working again:

I have created a PR as you suggested.

Best regards

Hi @heurtemattes,

Meantime, would it be possible to reduce build history size for all jobs to something like 5? we would like to transfer this instance to the new backend storage.

We updated our builds according to what you suggested.

closed

Hello @pstankie @heurtemattes, we're currently experiencing some issues with our Jenkins instance. Only one executor is currently available and builds are piling up.

Should be ok now.

Thanks!

Hello @pstankie @heurtemattes, we're experiencing a severely degraded Jenkins instance currently. The UI is barely responsive and build fail to trigger. Is there something you can do on your end?

Sometimes it just errors out

We are experiencing an infrastructure outage, we are working actively to address the issue.

Thanks. Since this morning the issues with the maven repo are solved. Unfortunately, for some reason our Jenkins CI is working with only one executor:

Best regards

Should be back now.

Thanks. Appreciated

The problem is still there. I'm doing some tests to understand exactly what is happening to this instance.

I think I found the issue with the kubernetes plugin, I will open a ticket upstream.

removed statewip label