SSH authentication errors in Jenkins for CDT.cloud Blueprint

Summary

CDT.cloud Blueprint is a fork of Theia Blueprint and uses mostly the same Jenkins configuration etc.

Although using the same set of configurations, the ssh authentication seems to fail. I'm unsure how to proceed here as it looks to be correctly configured. Can you take a look and check whether you can spot the problem?

Steps to reproduce

Run the Jenkins build: https://ci.eclipse.org/cdt-cloud/job/Jenkins%20Test/

What is the current bug behavior?

The offical CDT.cloud Blueprint repository can be found here.

I forked the repository and execute the Jenkins build on the fork to not "pollute" the original repository with my temporary changes. Once the build works I'll migrate it to the official repository.

Forked repository: https://github.com/sdirix/cdt-cloud-blueprint
Build on Jenkins for the forked repository: https://ci.eclipse.org/cdt-cloud/job/Jenkins%20Test/

The build fails pretty fast because of the following error (full log here, Jenkinsfile here):

[ssh-agent] Using credentials genie.cloud (ssh://genie.cloud@projects-storage.eclipse.org)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
Executing sh script inside container theia-dev of pod jenkins-test-master-4-cmw0d-sjwgs-crjvw
Executing command: "ssh-agent" 
exit
SSH_AUTH_SOCK=/tmp/ssh-PCwp47GhSmxx/agent.2849; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2850; export SSH_AGENT_PID;
echo Agent pid 2850;
SSH_AUTH_SOCK=/tmp/ssh-PCwp47GhSmxx/agent.2849
SSH_AGENT_PID=2850
Running ssh-add (command line suppressed)
ssh_askpass: exec(/home/jenkins/agent/workspace/Jenkins_Test_master@tmp/askpass_5882503119101644849.sh): No such file or directory

However the same call succeeds in the original Theia Blueprint build (full log here, Jenkinsfile here):

[ssh-agent] Using credentials genie.theia (ssh://genie.theia@projects-storage.eclipse.org)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
Executing sh script inside container theia-dev of pod theia2-master-54-s1xk7-cp79q-08jjn
Executing command: "ssh-agent" 
exit
SSH_AUTH_SOCK=/tmp/ssh-a27g0iMATwUJ/agent.2760; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2761; export SSH_AGENT_PID;
echo Agent pid 2761;
SSH_AUTH_SOCK=/tmp/ssh-a27g0iMATwUJ/agent.2760
SSH_AGENT_PID=2761
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/agent/workspace/Theia2_master@tmp/private_key_517335532415331484.key (/home/jenkins/agent/workspace/Theia2_master@tmp/private_key_517335532415331484.key)
[ssh-agent] Started.

The diff between the Jenkins files is minimal, click to expand

diff --git a/Jenkinsfile b/Jenkinsfile
index 4bf63e1..2aae10c 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -1,5 +1,5 @@
 /**
- * This Jenkinsfile builds Theia across the major OS platforms
+ * This Jenkinsfile builds CDT.cloud Blueprint across the major OS platforms
  */
 import groovy.json.JsonSlurper
 
@@ -217,13 +217,13 @@ spec:
                         container('theia-dev') {
                             script {
                                 signInstaller('exe', 'windows')
-                                updateMetadata('TheiaBlueprint.exe', 'latest.yml')
+                                updateMetadata('CDTCloudBlueprint.exe', 'latest.yml')
                             }
                         }
                         container('jnlp') {
                             script {
                                 uploadInstaller('windows')
-                                copyInstaller('windows', 'TheiaBlueprint', 'exe')
+                                copyInstaller('windows', 'CDTCloudBlueprint', 'exe')
                             }
                         }
                     }
@@ -280,7 +280,7 @@ def notarizeInstaller(String ext) {
     List installers = findFiles(glob: "${distFolder}/*.${ext}")
 
     if (installers.size() == 1) {
-        String response = sh(script: "curl -X POST -F file=@${installers[0].path} -F \'options={\"primaryBundleId\": \"eclipse.theia\", \"staple\": true};type=application/json\' ${service}/notarize", returnStdout: true)
+        String response = sh(script: "curl -X POST -F file=@${installers[0].path} -F \'options={\"primaryBundleId\": \"cdtcloud.blueprint\", \"staple\": true};type=application/json\' ${service}/notarize", returnStdout: true)
 
         def jsonSlurper = new JsonSlurper()
         def json = jsonSlurper.parseText(response)
@@ -314,12 +314,12 @@ def uploadInstaller(String platform) {
         def packageJSON = readJSON file: "package.json"
         String version = "${packageJSON.version}"
         sshagent(['projects-storage.eclipse.org-bot-ssh']) {
-            sh "ssh genie.theia@projects-storage.eclipse.org rm -rf /home/data/httpd/download.eclipse.org/theia/${version}/${platform}"
-            sh "ssh genie.theia@projects-storage.eclipse.org mkdir -p /home/data/httpd/download.eclipse.org/theia/${version}/${platform}"
-            sh "scp ${distFolder}/*.* genie.theia@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/theia/${version}/${platform}"
-            sh "ssh genie.theia@projects-storage.eclipse.org rm -rf /home/data/httpd/download.eclipse.org/theia/latest/${platform}"
-            sh "ssh genie.theia@projects-storage.eclipse.org mkdir -p /home/data/httpd/download.eclipse.org/theia/latest/${platform}"
-            sh "scp ${distFolder}/*.* genie.theia@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/theia/latest/${platform}"
+            sh "ssh genie.cloud@projects-storage.eclipse.org rm -rf /home/data/httpd/download.eclipse.org/cloud/blueprint/${version}/${platform}"
+            sh "ssh genie.cloud@projects-storage.eclipse.org mkdir -p /home/data/httpd/download.eclipse.org/cloud/blueprint/${version}/${platform}"
+            sh "scp ${distFolder}/*.* genie.cloud@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/cloud/blueprint/${version}/${platform}"
+            sh "ssh genie.cloud@projects-storage.eclipse.org rm -rf /home/data/httpd/download.eclipse.org/cloud/blueprint/latest/${platform}"
+            sh "ssh genie.cloud@projects-storage.eclipse.org mkdir -p /home/data/httpd/download.eclipse.org/cloud/blueprint/latest/${platform}"
+            sh "scp ${distFolder}/*.* genie.cloud@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/cloud/blueprint/latest/${platform}"
         }
     } else {
         echo "Skipped upload for branch ${env.BRANCH_NAME}"
@@ -331,7 +331,7 @@ def copyInstaller(String platform, String installer, String extension) {
         def packageJSON = readJSON file: "package.json"
         String version = "${packageJSON.version}"
         sshagent(['projects-storage.eclipse.org-bot-ssh']) {
-            sh "ssh genie.theia@projects-storage.eclipse.org cp /home/data/httpd/download.eclipse.org/theia/latest/${platform}/${installer}.${extension} /home/data/httpd/download.eclipse.org/theia/latest/${platform}/${installer}-${version}.${extension}"
+            sh "ssh genie.cloud@projects-storage.eclipse.org cp /home/data/httpd/download.eclipse.org/cloud/blueprint/latest/${platform}/${installer}.${extension} /home/data/httpd/download.eclipse.org/cloud/blueprint/latest/${platform}/${installer}-${version}.${extension}"
         }
     } else {
         echo "Skipped copying installer for branch ${env.BRANCH_NAME}"

The credentials seem to be there:

| projects-storage.eclipse.org-bot-ssh | ssh://genie.cloud@projects-storage.eclipse.org |

What is the expected correct behavior?

The ssh-agent should be successfully configured

Relevant logs and/or screenshots

Added links to all relevant files and logs above.

Edited May 05, 2022 by Frederic Gurr