jgit-publish-site job cannot deploy to project storage anymore

Summary

The jgit-publish-site job cannot deploy to project storage anymore.

Steps to reproduce

Run this job for the latest jgit release v6.5.0.202303070854-r

What is the current bug behavior?

Deployment fails with the error: Caused by: com.jcraft.jsch.JSchException: reject HostKey: projects-storage.eclipse.org

What is the expected correct behavior?

Deployment succeeds. Looks like something changed regarding this ssh key.

Relevant logs and/or screenshots

see https://ci.eclipse.org/jgit/job/stable/job/jgit-publish-site/176/console

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

It's blocking deployment of the JGit Maven site (most notably the javadoc) for the new release 6.5.0.202303070854-r shipped with the simultaneous release 2023-03. Hence the link org.eclipse.jgit JavaDoc on the project website documentation page returns "Not found".

changed the description

added IT-prioritymedium IT-severitynormal ~24521 team:releng labels

assigned to @fgurr

reject HostKey issue has been fixed by adding projects-storage.eclipse.org to the known_hosts file.

Build fails with a test error now.

closed

Thanks for your help.

I ran the job once more and all tests succeeded. But now defployment of the site failed with this error:

14:10:33  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-site-plugin:4.0.0-M4:deploy (default-cli) on project org.eclipse.jgit-parent: Error uploading site
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:306)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
14:10:33      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
14:10:33      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
14:10:33      at java.lang.Thread.run (Thread.java:829)
14:10:33  Caused by: org.apache.maven.plugin.MojoExecutionException: Error uploading site
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.push (AbstractDeployMojo.java:454)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:311)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.deployTo (AbstractDeployMojo.java:294)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.execute (AbstractDeployMojo.java:169)
14:10:33      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
14:10:33      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
14:10:33      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
14:10:33      at java.lang.Thread.run (Thread.java:829)
14:10:33  Caused by: org.apache.maven.wagon.authentication.AuthenticationException: Cannot connect. Reason: Auth cancel
14:10:33      at org.apache.maven.wagon.providers.ssh.jsch.AbstractJschWagon.openConnectionInternal (AbstractJschWagon.java:281)
14:10:33      at org.apache.maven.wagon.AbstractWagon.openConnection (AbstractWagon.java:139)
14:10:33      at org.apache.maven.wagon.AbstractWagon.connect (AbstractWagon.java:239)
14:10:33      at org.apache.maven.wagon.AbstractWagon.connect (AbstractWagon.java:176)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.push (AbstractDeployMojo.java:418)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:311)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.deployTo (AbstractDeployMojo.java:294)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.execute (AbstractDeployMojo.java:169)
14:10:33      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
14:10:33      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
14:10:33      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
14:10:33      at java.lang.Thread.run (Thread.java:829)
14:10:33  Caused by: com.jcraft.jsch.JSchException: Auth cancel
14:10:33      at com.jcraft.jsch.Session.connect (Session.java:518)
14:10:33      at com.jcraft.jsch.Session.connect (Session.java:183)
14:10:33      at org.apache.maven.wagon.providers.ssh.jsch.AbstractJschWagon.openConnectionInternal (AbstractJschWagon.java:267)
14:10:33      at org.apache.maven.wagon.AbstractWagon.openConnection (AbstractWagon.java:139)
14:10:33      at org.apache.maven.wagon.AbstractWagon.connect (AbstractWagon.java:239)
14:10:33      at org.apache.maven.wagon.AbstractWagon.connect (AbstractWagon.java:176)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.push (AbstractDeployMojo.java:418)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:311)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.deployTo (AbstractDeployMojo.java:294)
14:10:33      at org.apache.maven.plugins.site.deploy.AbstractDeployMojo.execute (AbstractDeployMojo.java:169)
14:10:33      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
14:10:33      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
14:10:33      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
14:10:33      at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
14:10:33      at java.util.concurrent.FutureTask.run (FutureTask.java:264)
14:10:33      at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
14:10:33      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
14:10:33      at java.lang.Thread.run (Thread.java:829)

reopened

marked the checklist item Medium as completed

marked the checklist item Urgent as incomplete

We're still investigating.

I've taken a look at the system logs and I can't find a matching entry for the Jgit bot in the time period mentioned in the log fragment above. I also wasn't able to find which job(https://ci.eclipse.org/jgit/job/stable/job/jgit-publish-site/) the above logs came from in order to try and cross reference things.

Am I missing something?

I found this error in the following build:

https://ci.eclipse.org/jgit/job/stable/job/jgit-publish-site/178/console

Thanks.

The logs don't seem to have anything relevant and I'm not seeing that error repeated for follow up builds.

The issue seems to be limited to the JGit Jenkins instance. Accessing projects-storage.eclipse.org with the JGit credentials from another Jenkins instance works fine.

• SSH was not happy with finding two host keys (RSA and ECDSA) for the same server in the known_hosts file and was just hanging in my simple test job (after my initial change)
• With only the ECDSA key in place, the simple test job is succeeding again
• the jgit-publish-site job fails with JSch's non-descriptive Auth cancel error
• But I found this in the logs:

[INFO] --- maven-site-plugin:4.0.0-M4:deploy (default-cli) @ org.eclipse.jgit-parent ---
Password for genie.jgit@projects-storage.eclipse.org:

It seems like the maven-site-plugin is trying to use password based authentication (or failing public key authentication and falling back).

It looks like the private key (and passphrase) to use by Maven Wagon SSH can be set in settings.xml, see https://maven.apache.org/guides/mini/guide-deployment-security-settings.html

Do I have access to that or can you put the key there for me?

ping ?

Adding the SSH credentials into the Maven settings.xml is less than ideal.

I would like to understand "how this has ever worked before?"(TM). What changed between December 7, 2022 and February 21, 2023?

during that time we updated maven plugins in https://git.eclipse.org/r/c/jgit/jgit/+/199268

• maven-deploy-plugin was updated from 3.0.0-M1 to 3.0.0
• maven-site-plugin was updated from 3.9.1 to 4.0.0-M4
• wagon-ssh was updated from 3.5.1 to 3.5.2 on 2022-10-10: https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+blame/refs/heads/master/pom.xml#344

I found this note on https://maven.apache.org/wagon/wagon-providers/wagon-ssh/

"This Wagon provider is deprecated and will be removed in version 4.0.0".

Maybe there is some interference from updating the maven-site-plugin to 4.0.0. Though I think wagon-ssh is used by the maven-deploy-plugin

The difference between wagon 3.5.1 and 3.5.2 looks innocent regarding ssh authentication: https://github.com/apache/maven-wagon/compare/wagon-3.5.1...wagon-3.5.2

Is it possible to switch to the ssh-external wagon provider for the maven-site-plugin? IIUC this might work with ssh-agent (see https://maven.apache.org/plugins/maven-deploy-plugin/examples/deploy-ssh-external.html).

I tried switching to wagon-ssh-external, this failed with the error that protocol scp is not supported: https://ci.eclipse.org/jgit/job/stable/job/jgit-publish-site/189/console next I will try to downgrade the maven-site-plugin to 3.12.1

with maven-site-plugin 3.12.1 this failed with the same error https://ci.eclipse.org/jgit/job/stable/job/jgit-publish-site/190/console

added statetodo label

marked this issue as blocked by #3137 (closed)