Jersey JIPP has no executors to run jobs

Jersey CI Jenkins executors starved to death. See https://ci.eclipse.org/jersey.

added IT-severitynormal ~24521 team:releng labels

Another Friday, another executor starvation... Do the executors have free weekends, too?

assigned to @fgurr

added IT-prioritymedium label

I did manually restart the Jersey JIPP each time (forgot to comment on this ticket last week).

Two occasions might not be a pattern yet, but are you running any particular build jobs only on Fridays?

Now is Monday and again no executors...

https://ci.eclipse.org/jersey/

Not only on Friday, I'd say the executors are killed when more jobs are started and waiting for executors. Usually, we see something like the following when the executors are killed (not sure if that is the cause, or consequence):

Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[continuous-integration/3.x/39:continuous-integration/3.x #39 (closed)] but FlowNode was not found in storage for head id:FlowNodeId 1:30 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:701) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:560) at hudson.model.RunMap.retrieve(RunMap.java:233) at hudson.model.RunMap.retrieve(RunMap.java:61) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:650) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:632) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:530) at hudson.model.RunMap.getById(RunMap.java:213) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:949) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:961) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getExecution(CpsStepContext.java:218) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:98) at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:264) at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:262) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:840)

I've restarted the JIPP again. This is probably still related to #682. I will keep a closer look.

mentioned in issue #2111 (closed)

closed

this is another Friday's run out of executors:

https://ci.eclipse.org/jersey/job/continuous-integration/job/PR-5175/18/console

could you please take a look?

reopened

Jenkins was restarted. Builds are running again.

closed