[temurin-compliance] Solaris JCK machines require Git Upgrade to Git 2.31.0

Summary

AQA Tests fail to run correctly on Solaris 10 Servers.

Steps to reproduce

Execute an AQA Test run on a solaris machine.

What is the current bug behavior?

An error is returned when the following command is run

git clone --depth 1 --reference-if-able /export/home/jenkins//openjdk_cache -b master https://github.com/eclipse-openj9/openj9.git error: unknown option `reference-if-able'

What is the expected correct behavior?

Git should clone the repository without issue.

Relevant logs and/or screenshots

Errors can be seen here : https://ci.eclipse.org/temurin-compliance/view/JCK8/job/Test_openjdk8_hs_sanity.jck_x86-64_solaris/34/console

10:15:20  TKG_REPO=https://github.com/adoptium/TKG.git
10:15:20  TKG_BRANCH=cb24260d36e0a118e2847f946dc0267548ba6de6
10:15:20  get functional test material...
10:15:20  git clone --depth 1 --reference-if-able /export/home/jenkins//openjdk_cache -b master https://github.com/eclipse-openj9/openj9.git
10:15:20  error: unknown option `reference-if-able'
10:15:20  usage: git clone [options] [--] <repo> [<dir>]
10:15:20  
10:15:20      -v, --verbose         be more verbose
10:15:20      -q, --quiet           be more quiet

Some relevant/useful information, in order to compile Git 2.31.0 for Solaris 10, it needs to be compiled from source.. below are an example from the Adoptium ansible playbook for performing the tasks required. This PR provides details on what has been done in the Adoptium infrastructure playbooks to achieve this. https://github.com/adoptium/infrastructure/pull/2665

https://github.com/adoptium/infrastructure/blob/master/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/Common/tasks/Solaris.yml

- name: Test if nonCSW Git Is Installed At /usr/local/bin
  shell: /usr/local/bin/git --version >/dev/null
  failed_when: false
  register: git_noncsw_installed
  changed_when: false
  tags:
    - build_tools

- name: Test if git is installed at the correct version
  shell: /usr/local/bin/git --version | sed -e 's/git version //g' | awk -F'[.]' '{print $1 "." $2}'
  when: git_noncsw_installed.rc == 0
  register: git_noncsw_version
  changed_when: false
  tags:
    - build_tools

- name: Set python27
  set_fact:
    ansible_python_interpreter: /opt/csw/bin/python
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))
- name: Get Git Source
  get_url:
    url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.31.0.tar.gz
    dest: /tmp/git-2.31.0.tar.gz
    mode: 0440
    checksum: sha256:bc6168777883562569144d536e8a855b12d25d46870d95188a3064260d7784ee
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))

- name: Extract git 2.31.0 source
  shell: gzip -cd git-2.31.0.tar.gz | tar xpf -
  args:
    chdir: /tmp
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))
  tags:
    - build_tools

- name: Create Git Temporary Compilation Library directory
  file:
    path: /tmp/git-2.31.0/libexec/git-core
    state: directory
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))
  tags:
    - build_tools

- name: Copy Existing Git Remote HTTPS Library Into Compile Area
  copy:
    src: "/opt/csw/libexec/git-core/git-remote-https"
    dest: "/tmp/git-2.31.0/libexec/git-core/"
    remote_src: yes
    mode: 0775
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))
  tags:
    - build_tools

- name: Compile Git 2.31.0
  shell: |
    cd /tmp/git-2.31.0
    ./configure --with-curl=/opt/csw/lib/libcurl.so --with-openssl=/opt/csw/lib/libssl.so --with-expat
    gmake -j4
    gmake install
  environment:
    PATH: /opt/solarisstudio12.3/bin/:/opt/csw/bin/:/usr/ccs/bin:/usr/bin
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))
  tags:
    - build_tools

- name: Copy New Git Remote HTTPS Library Into Runtime Area
  copy:
    src: "/tmp/git-2.31.0/libexec/git-core/git-remote-https"
    dest: "/usr/local/bin/"
    remote_src: yes
    mode: 0775
  when: (git_noncsw_installed.rc != 0) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt'))
  tags:
    - build_tools

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

(What is the impact of this issue? Is it blocking a release? Are there any time constraints?, for example: "We have a release tomorrow")

added IT-priorityhigh label

added IT-prioritymedium label and removed IT-priorityhigh label

added IT-priorityhigh label and removed IT-prioritymedium label

added IT-severitynormal Servers labels

changed title from Solaris JCK machines require Git Upgrade to Git 2.31.0 to [temurin-compliance] Solaris JCK machines require Git Upgrade to Git 2.31.0

added statetodo label

assigned to @fgurr

added statewip label and removed statetodo label

• siteox-solaris10u11-sparcv9-1
• equinix_esxi-solaris10u11-x64-1
• equinix_esxi-solaris10u11-x64-2
• equinix_esxi-solaris10-x64-1

Is there a good reason why the default git installation still points to version 2.3.1? Is this expected?

• Should /usr/bin/git be symlinked to /usr/local/bin/git?
• Or should /usr/local/bin be included in PATH? (see also #2276 (comment 1051304) => so this is not a problem for Jenkins jobs, since PATH is set correctly in the Jenkins agent config)

bash-3.2# git --version 
git version 2.3.1
bash-3.2# which git
/opt/csw/bin/git

bash-3.2# /usr/local/bin/git --version
git version 2.31.0

Please test and reopen this ticket if something is missing.

closed

removed statewip label

@fgurr would you mind altering the paths on the solaris jenkins agents, so that the /usr/local/bin/git takes precedence, Ive had a look at the jenkins configuration, and they also seem to have a default git path set in tools location, I think we'll need to switch that also.

reopened

I have changed the tools location for Git in the agent configurations.

Adjusting the PATH environment variables could lead to unintended side effects, since other tools are installed in /opt/csw/bin as well.

Let me know if changing the tools location in the agent config is sufficient.

added team:releng label

@fgurr Thanks, I think we'll need to change the path, we've tested this configuration within the project, so I'd expect it to be ok. The current change to just the tool path is causing an error.. where it cant find remote-https

11:40:13  The recommended git tool is: /usr/local/bin/git
11:40:13  No credentials specified
11:40:13  Cloning the remote Git repository
11:40:13  ERROR: Error cloning remote repo 'origin'
11:40:13  hudson.plugins.git.GitException: Command "/usr/local/bin/git fetch --tags --force --progress -- https://github.com/adoptium/aqa-tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
11:40:13  stdout: 
11:40:13  stderr: git: 'remote-https' is not a git command. See 'git --help'.

added statereview label

I will look into it.

I've changed the order in the PATH variable in the agent configurations from

/opt/csw/bin:/usr/bin:/bin:/usr/local/bin:/usr/sbin

to

/usr/local/bin:/opt/csw/bin:/usr/bin:/bin:/usr/sbin

/usr/local/bin contains the following, so I guess this should be fine:

ls -l /usr/local/bin
total 260060
lrwxrwxrwx   1 root     root          36 Oct 20  2021 ant -> /usr/local/apache-ant-1.10.5/bin/ant
-rwxr-xr-x 137 root     staff    28913288 Dec  6 17:01 git
-rwxr-xr-x   2 root     staff     163107 Dec  6 17:01 git-cvsserver
-rwxr-xr-x 137 root     staff    28913288 Dec  6 17:01 git-receive-pack
-rwxrwxr-x   1 root     root     1664400 Mar  5  2015 git-remote-https
-rwxr-xr-x   2 root     staff    14655140 Dec  6 17:01 git-shell
-rwxr-xr-x 137 root     staff    28913288 Dec  6 17:01 git-upload-archive
-rwxr-xr-x 137 root     staff    28913288 Dec  6 17:01 git-upload-pack
-rwxr-xr-x   1 root     staff     401052 Dec  6 17:01 gitk
lrwxrwxrwx   1 root     root          37 Oct 20  2021 mvn -> /usr/local/apache-maven-3.6.3/bin/mvn

For good measure I've also set:

ln -s /usr/local/bin/git /usr/bin/git

mentioned in issue #2276 (closed)

closed

reopened

Hi @fgurr still no joy..

17:05:46  Cloning repository https://github.com/adoptium/aqa-tests.git
17:05:47   > /usr/local/bin/git init /export/home/jenkins/workspace/Test_openjdk8_hs_sanity.jck_x86-64_solaris/aqa-tests # timeout=10
17:05:47  [WARNING] Reference path does not exist: /export/home/jenkins//openjdk_cache
17:05:47  Fetching upstream changes from https://github.com/adoptium/aqa-tests.git
17:05:47   > /usr/local/bin/git --version # timeout=10
17:05:47  ERROR: Error cloning remote repo 'origin'
17:05:47  hudson.plugins.git.GitException: Command "/usr/local/bin/git fetch --tags --force --progress -- https://github.com/adoptium/aqa-tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
17:05:47  stdout: 
17:05:47  stderr: git: 'remote-https' is not a git command. See 'git --help'.

Please always provide the URL to the failing build.

Can I ask if when you compiled the new version of git, you copied the previously installed version of git-remote-https into the compilation error...

This is the relevant section of the ansible playbook..

- name: Get Git Source
  get_url:
    url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.31.0.tar.gz
    dest: /tmp/git-2.31.0.tar.gz
    mode: 0440
    checksum: sha256:bc6168777883562569144d536e8a855b12d25d46870d95188a3064260d7784ee
  when: (git_noncsw_installed.rc != 0 ) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt') )

- name: Extract git 2.31.0 source
  shell: gzip -cd git-2.31.0.tar.gz | tar xpf -
  args:
    chdir: /tmp
  when: (git_noncsw_installed.rc != 0 ) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt') )
  tags:
    - build_tools

- name: Create Git Temporary Compilation Library directory
  file:
    path: /tmp/git-2.31.0/libexec/git-core
    state: directory
  when: (git_noncsw_installed.rc != 0 ) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt') )
  tags:
    - build_tools

- name: Copy Existing Git Remote HTTPS Library Into Compile Area
  copy:
    src: "/opt/csw/libexec/git-core/git-remote-https"
    dest: "/tmp/git-2.31.0/libexec/git-core/"
    remote_src: yes
    mode: 0775
  when: (git_noncsw_installed.rc != 0 ) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt') )
  tags:
    - build_tools

- name: Compile Git 2.31.0
  shell: |
    cd /tmp/git-2.31.0
    ./configure --with-curl=/opt/csw/lib/libcurl.so --with-openssl=/opt/csw/lib/libssl.so --with-expat
    gmake -j4
    gmake install
  environment:
    PATH: /opt/solarisstudio12.3/bin/:/opt/csw/bin/:/usr/ccs/bin:/usr/bin
  when: (git_noncsw_installed.rc != 0 ) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt') )
  tags:
    - build_tools

- name: Copy New Git Remote HTTPS Library Into Runtime Area
  copy:
    src: "/tmp/git-2.31.0/libexec/git-core/git-remote-https"
    dest: "/usr/local/bin/"
    remote_src: yes
    mode: 0775
  when: (git_noncsw_installed.rc != 0 ) or (git_noncsw_installed.rc == 0 and git_noncsw_version.stdout is version_compare('2.31', operator='lt') )
  tags:
    - build_tools

Can I ask if when you compiled the new version of git, you copied the previously installed version of git-remote-https into the compilation error...

I'm not sure what that means, but I did not copy any executables.

Ok, this may be the issue.. as part of the process of compiling Git 2.31.0 on solaris 10, its necessary to copy the git-remote-https into the compile directory, e.g..

unzip the git2.31.0 source bundle.. which should have this path in it /tmp/git-2.31.0/libexec/git-core I then had to copy the existing git-remote-https into the extracted source directory . .e.g

SRC : ( /opt/csw/libexec/git-core/git-remote-https )
DST : ( /tmp/git-2.31.0/libexec/git-core/ )

I then had to do the configure, make and install.. ( again , this assume the 2 libraries for openssl are in /opt/csw/lib

    cd /tmp/git-2.31.0
    ./configure --with-curl=/opt/csw/lib/libcurl.so --with-openssl=/opt/csw/lib/libssl.so --with-expat
    gmake -j4
    gmake install

Finally I had to copy the recompiled version of git-remote-https into /usr/lib/bin

src: "/tmp/git-2.31.0/libexec/git-core/git-remote-https"
    dest: "/usr/local/bin/"

As you can see in my comment above (#2145 (comment 1054229)), there is a git-remote-https executable in the /usr/local/bin dir already. How can I tell if that's the right executable or if the executable is "flawed"?

So far I've not been able to reproduce the clone error on the Solaris machines. I was able to clone via the build-in SCM mechanism and on the command line just fine.

Ok. I'm a step closer. Checkout works fine with Freestyle and Matrix jobs. It fails with pipeline jobs. I can reproduce the error now.

FTR: I did install Git 2.31.0 on the Solaris machines by applying the AdoptOpenJDK_UNIX_Playbook which contains the exact tasks that are listed in Scott's comment above (#2145 (comment 1054248)). So the git-remote-https executable should be alright.

Right now, I'm suspecting an issue with the pipeline part of the Git plugin.

I've tested it with another Jenkins instance that is running on LTS version 2.361.4 and the latest plugin updates, but it did not make a difference. I've changed the order of the entries in PATH, but it did not seem to help.

For now, I don't have a solution, only a confirmed workaround: using a shell script step instead of the build in git or checkout pipeline steps. I've also tried to use a ssh repo URL but get another odd error:

stderr: command-line: line 0: unknown configuration option: SendEnv
terminating, 1 bad configuration option(s)

@scottfryer & @saddisonxls: does this combination (git clone from https URL on Solaris) work on the external TC Jenkins instance?

Scott set up the external machines and I believe they're still working ok.

Can you try setting up the TC CI to have export PATH=/usr/local/bin:$PATH & as the Prefix Start Agent Command on the agent setup in the jenkins UI to force the correct git tools picked up?

Does not help, unfortunately.

at this point , is it worth replacing the 2.3.1 version of Git with the upgraded version, so that only a single version is available? @saddisonxls what do you think ?

Hi @fgurr couple of things we can try out...

The git-remote-https does work as my user (scottfryer) when I do the following export PATH=/usr/local/bin:$PATH git clone --depth 1 --reference-if-able /export/home/jenkins//openjdk_cache -b master https://github.com/eclipse-openj9/openj9.git

My default path did not include this, but this is my path now it works.. /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/csw/bin:/opt/csw/bin/

A config change we could try in jenkins is to alter the way it launches the agent..

In the advanced config setting for the node, alter the Prefix Start Agent Command to include the following at the beginning:

export PATH=/usr/local/bin:$PATH

Thanks @scottfryer, that seems to have fixed it. I had concatenated the two export commands by a semicolon instead of && and that did not work.

I've set the Prefix Start Agent Command for all Solaris agents.

Please test it.

@fgurr Im just testing it on x86, would you mind checking jck-siteox-solaris10u11-sparcv9-1 , as it appears to still have the issue : https://ci.eclipse.org/temurin-compliance/view/Solaris/job/Test_openjdk8_hs_sanity.jck_sparcv9_solaris/62/console

jck-siteox-solaris10u11-sparcv9-1 required a reconnect.

@fgurr ,the x86 is working! ,

15:36:06 git clone --depth 1 --reference-if-able /export/home/jenkins//openjdk_cache -b master https://github.com/eclipse-openj9/openj9.git 15:36:06 Cloning into 'openj9'... 15:36:06 info: Could not add alternate for '/export/home/jenkins//openjdk_cache': reference repository '/export/home/jenkins//openjdk_cache' is not a local repository. 15:38:28 Updating files: 3% (336/10170) Updating files: 4% (407/10170) Updating files: 5% (509/10170) Updating files: 6% (611/10170) Updating files: 7% (712/10170)

I'll retest sparc... and then if it works close this issue, thanks for all the help and perseverance on this one!

and the sparc is now working also.

closed

Hi Fred, we're still having one issue with the updated version of git on the solaris x64 box, would you mind adding this to the node config ( https://ci.eclipse.org/temurin-compliance/computer/jck-equinix_esxi-solaris10-x64-1/configure ), and then stopping/restarting it so I can retest ahead of next weeks release

Prefix Start Agent Command export PATH=/usr/local/bin:$PATH &&

Done.

Working fine now, thanks!

reopened

closed