Signing OS X application less stable than it used to be - timing out when trying to sign

We have a staging version of the MacOS signing service in our cluster running. Could you give this a try? The url is https://cbi-staging.eclipse.org/macos/codesign/sign instead of the production url that uses cbi.eclipse.org. The rest is the same.

This staging server should be much more stable and reliable as the production one. We wanted to switch to the new version already but could not find anybody willing to test the staging service so far.

Some background: the production version uses Apple's own signing software that needs to run on a dedicated macOS machine. The staging version uses a different open-source signing software that is capable of running in a kubernetes / openshift cluster.

The EPP should be a good place to test this - and now is a good time to do that because the next full release isn't until March, but we do have a bunch of milestone releases. See https://github.com/eclipse-packaging/packages/pull/255 - we run builds typically once per day, and each build signs 20 macOS Eclipse.apps ranging from 300-600MB. So I guess this is a good test for you.

that would be amazing. The new version also supports deep signing of dmg files. Something that you had to workaround with the previous version, so that hopefully simplifies the signing for our projects.

I could not test it myself, but if you can point me to some files / workflows / jobs that do the signing I can also look at it.

I ran it and got a server error (job https://ci.eclipse.org/packaging/job/epp/job/master/269) - Have restarted it with an attempt to archive all *.log files as artifacts.

16:14:29  [INFO] --- eclipse-macsigner-plugin:1.5.0:sign (sign) @ epp.package.committers ---
16:15:01  [INFO] [Tue Dec 10 21:14:59 GMT 2024] Signing OS X application '/home/jenkins/agent/workspace/epp_master/packages/org.eclipse.epp.package.committers.product/target/products/epp.package.committers/macosx/cocoa/x86_64/Eclipse.app'...
16:15:58  [WARNING] [Tue Dec 10 21:15:52 GMT 2024] HTTP request failed.
16:15:58  HTTP Error 500 (reason: Server Error)
16:15:58  HTTP Error 500
16:15:58  Problem accessing '/macos/codesign/sign'
16:15:58  Reason: Server Error
16:15:58  Server response has been saved to '/home/jenkins/agent/workspace/epp_master/packages/org.eclipse.epp.package.committers.product/target/products/epp.package.committers/macosx/cocoa/x86_64/Eclipse.app_11567459399641459692.zip-15375784280457084210-OSXAppSigner.log'

I could not test it myself, but if you can point me to some files / workflows / jobs that do the signing I can also look at it.

https://ci.eclipse.org/packaging/job/epp/job/master/ builds all the EPPs and signs them. https://ci.eclipse.org/packaging/job/notarize-downloads/ is used to notarize the results. We only notarize published builds, but when epp/job/master completes I will kick off the notarization and then I'll test on an M1 macbook I have access to.

got this from the logs:

non-nested file is a Mach-O binary; signing accordingly Contents/Eclipse/plugins/com.sun.jna_5.15.0.v20240915-2000/com/sun/jna/AltCallingConvention.class
signing Mach-O file Contents/Eclipse/plugins/com.sun.jna_5.15.0.v20240915-2000/com/sun/jna/AltCallingConvention.class
Error: binary parsing error: Buffer is too short for 52 arches

will check if this is a known problem with the tool.

https://github.com/indygreg/apple-platform-rs/issues/170

Coincidentially, the guy was signing the same jar. What are the odds?

will update the service with the suggested fix.

ok updated version with the fix is deployed, can you restart the job?

I have restarted it now (feel free to start that build if you want - I will next be able to see your messages tomorrow afternoon Ottawa time).

so I downloaded eclipse-committers-2025-03-M1-macosx-cocoa-x86_64.tar.gz from the projects storage and tested signing it locally. Without the option, I get the same error. Adding the option makes the signing successful.

I double checked that the latest deployment of the staging service includes the fix (I did it wrongly yesterday evening, it was late and I forget how the deployment script works).

Dont know how to restart the jenkins build, but if you trigger a new one it should work imho.

I also investigated the failures that you have with the production service and so far I could not figure out what the problem is as I dont see anything in the logs yet.

I have the service more memory as it was set to -Xmx512m but the file you you wanna sign is > 500MB. It is transferred as part of a post request, so not sure how that is handled internally by the service.

In any case, I gave it 2GB of memory.

I kicked it off again.

so I watched this time the backend server and see whats happening.

but no request was hitting the actual service yet (the lines before are from a monitoring job):

the problem might be related to some kubernetes / openshift timeouts as the 504 is certainly not coming from the service itself:

https://github.com/kubernetes/ingress-nginx/issues/3976

Is it possible that the files that have to be signed have increased in size considerably the last couple of runs?

@fgurr As I cant access this information, does your team have experience with something like that? What are our nginx timeout settings? Can this be adjusted per service or cluster / namespace wide? From the build it looks like that the request fails after around 60s with the 504 error:

Apparently you can do that in openshift:

Hi, just for benefit of folks working with Openshift : Instead of ingress, I created an openshift Route. No YAML needed, they had a straightforward wizard in the openshift console. After creating, in its settings there are "Annotations". I added: key: haproxy.router.openshift.io/timeout , value: 5m and saved. That increased my deployment's gateway timeout limit to 5 minutes instead of default 60 seconds.

oh I found this in the route config:

            "haproxy.router.openshift.io/timeout": "60s"

ok redeployed the service now with 600s timeout the same as we used for the forwarding service for the production service:

I kicked it off again.

Dont know how to restart the jenkins build, but if you trigger a new one it should work imho.

You can press the Build Now button on https://ci.eclipse.org/packaging/job/epp/job/master/ as shown in this screenshot. The first signing takes place approx ~30 minutes later

Build #275 completed the signing step successfully:

20:16:06  [INFO] --- eclipse-macsigner-plugin:1.5.0:sign (sign) @ epp.package.committers ---
20:16:44  [INFO] [Thu Dec 12 01:16:40 GMT 2024] Signing OS X application '/home/jenkins/agent/workspace/epp_master/packages/org.eclipse.epp.package.committers.product/target/products/epp.package.committers/macosx/cocoa/x86_64/Eclipse.app'...
20:18:51  [INFO] [Thu Dec 12 01:18:40 GMT 2024] Signing OS X application '/home/jenkins/agent/workspace/epp_master/packages/org.eclipse.epp.package.committers.product/target/products/epp.package.committers/macosx/cocoa/aarch64/Eclipse.app'...
20:20:12  [INFO]

But then the build failed because of a broken pipe while doing scp that seems unrelated:

20:26:19  + scp eclipse-committers-2025-03-M1-linux-gtk-riscv64.tar.gz eclipse-committers-2025-03-M1-linux-gtk-riscv64.tar.gz.md5 eclipse-committers-2025-03-M1-linux-gtk-riscv64.tar.gz.sha1 eclipse-committers-2025-03-M1-linux-gtk-riscv64.tar.gz.sha512 genie.packaging@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/technology/epp/building//
20:42:41  client_loop: send disconnect: Broken pipe
20:42:41  lost connection

I have kicked off the build again.

I dont have that button, but great that the signing succeeds now. The timeout was a stupid oversight, all the existing signing services use 60s for now, but that might be too low also for the windows signing service as the executables might also be that big there. For the jar signing it is ok though imho.

so I also did investigate why the current production service fails randomly atm.

We have actually 2 MacOS machines running that are selected randomly when signing a file. One of them seems to have run amok and we did not spot that in our monitoring jobs. The jobs failed, but resolved quickly afterwards as then the other machine was used. So I restarted the failing machine and it seems to run fine again.

Sorry for the inconvenience, but it might lead to the situation that we can finally move the staging service to production and then have everything running in the OpenShift cluster, avoiding such problems hopefully and being able to scale up the number of services if needed.

We had two back to back builds successful on the https://cbi-staging.eclipse.org/macos/codesign/sign service. I am running notarization now - the notarized files will be in https://download.eclipse.org/technology/epp/staging/ (the *.dmg files) soon.

@netomi the EPP was signed and notarized successfully. I have a macOS machine I can try it on later to ensure it looks correct, but if you have knowledge + access to correct hardware to check a dmg file or two that would be appreciated: https://download.eclipse.org/technology/epp/staging/

I asked my colleagues with access to a mac machine as I am lacking myself.

so my colleague could successfully run the following dmg file:

eclipse-committers-2025-03-M1-macosx-cocoa-x86_64.dmg

I inspected the signature on this file and it prints:

          - subject: 'CN=Developer ID Application: Eclipse Foundation, Inc. (JCDTMS22B4), OU=JCDTMS22B4, O=Eclipse Foundation, Inc., C=US'

so it looks good from my side.

I did the user flow on a mac book - I downloaded the dmg and it ran as expected without unexpected warnings/errors.

Thanks @netomi the service looks good now, correct output and build seems stable in this regard. Let me know when the new service is on the default URL so I can remove https://github.com/eclipse-packaging/packages/blob/af9001900277ac0feec7fa6ef2890b1b1190e477/releng/org.eclipse.epp.config/parent/product/pom.xml#L108-L109

great to hear and thanks for the patience. There is currently another project testing the new staging service and I will reach out to others as well that do use MacOSX signing a lot to make sure switching the service does not lead to some surprises.

added service:signing-mac team:security labels

mentioned in issue #5431

mentioned in issue #4258

mentioned in issue #4871 (closed)

@netomi after a month of stable operation the signing service has failed numerous times this week. Have a look at https://ci.eclipse.org/packaging/job/epp/job/master/355/ or other recently failed jobs.

I am still using the staging signer: https://github.com/eclipse-packaging/packages/blob/e66c4e9c066833914c083cf05ae8733da31d6f18/releng/org.eclipse.epp.config/parent/product/pom.xml#L108

hmm I could see some error in the access log that puzzles me. I restarted the service, can you restart a build?

Started https://ci.eclipse.org/packaging/job/epp/job/master/356/

It completed successfully.

ty for restarting the job. So I assume that the service went into a state it could not recover from anymore. The log files did not indicate something specific, will monitor and add some more alive checks and restart if needed.

Signing OS X application less stable than it used to be - timing out when trying to sign

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Priority

Severity

Impact

Designs

Child items ...

Activity