[automotive.score] hosting a bazel-remote cache

Summary

We would like to explore hosting a bazel-remote cache instance to significantly reduce Bazel build and test times across GitHub Actions workflows and developer machines.

bazel-remote can be configured as a lightweight HTTP/GRPC server with disk- or cloud-backed storage, using a simple YAML config file specifying cache directory, max size and access options.

It would be interesting to find out which hosting options we have at Eclipse.

Steps to reproduce

1. Use Bazel with local disk/repo cache only (as currently implemented in CI workflows).

2. Run repeated builds for targets (e.g., bazel test //...) in GitHub Actions.

3. Observe that even when local disk cache hits, the analysis and external fetch stages still take 2–3 minutes or more.

What is the current bug behavior?

Currently, we rely only on ephemeral local cache in CI. This:

• Does not persist across runners or job executions.
• Adds overhead to analysis/loading even when outputs don’t change.
• Leads to slow workflow execution (~3 min/test job vs. <10s locally).

What is the expected correct behavior?

CI runners and devs should be able to leverage a shared, persistent Bazel remote cache (bazel-remote or similar) to:

• Avoid redundant downloads of external dependencies.

• Reuse analysis/build/test outputs across executions and PRs.

• Reduce CI cost and execution time.

Relevant logs and/or screenshots

Analyzing: 3 targets (151 packages loaded, 1788 targets configured)
INFO: Analyzed 3 targets (152 packages loaded, 3692 targets configured)
...
INFO: Build completed, 1 test FAILED, 19 total actions

Total runtime: ~3m Expected with cache: <20s

Priority

• Urgent
• High
• Medium
• Low

Severity

• Blocker
• Major
• Normal
• Low

Impact

This has a significant impact on:

• CI latency and developer feedback loop time.
• Resource consumption and GitHub Actions costs ( though are repos are public)

added IT-prioritylow IT-severitylow service:GitHub statetodo team:releng labels

marked the checklist item Medium as completed

marked the checklist item Medium as incomplete

@skappel @gerdroid @grandyho @hartmannnico @thiloschmitt can you give an assessment on the impact of this issue?

• The possible build time duration improvement in the given example does sounds like a problem with a lower severity. Are there other examples where the build duration differences are more severe?
• As noted above, there are no additional cost for longer running GitHub action workflows.

@fgurr thank you for your note.

The build time difference is inherent to the design and working of the build system with the combination of being employed in isolated CI runs (which don't share any data or state). This is a well-known and well-understood problem. For that resaon bazel-remote caches were invented. (Side note: Similar issues exist with Yocto and aharing sstate). So I believe the proposed analysis and solution is sound.

I agree to your assessment on the insensitivity of CI cost regarding build time length. Still, longer build times delay feedback to contributors and maintainers, hence reducing iteration throughput. I'd welcome if we could tackle this, if feasible in terms of resources and general rules.

There is no doubt about the benefits of short build durations and reducing feedback delay.

I'm trying to get a better insight on how much this impacts the Score project. The given example shows a potential time saving of ~3 minutes. Are there any other builds with longer durations and potential for time savings (e.g. an hour-long build only takes 10 minutes with a cache)?

We are currently evaluating a self-hosted version of Develocity which features a built-in cache (https://docs.gradle.com/enterprise/bazel-configuration/#remote-cache). If you agree, we can give you credentials to test the cache functionality.

To set expectations: the evaluation phase for our Develocity instance will end on June 30, 2025. Based on community feedback and a few other factors, we will then decide whether we continue this service or not.

Okay, I see. Let me try to explain:

In the current state of the project build times are relatively low and differences with/without caching are immense on a theoretical level (e.g. 10x as described above), but practically it won't have an overly serious impact. That said, the project is very much in its infancy - at least in terms of its amount of source and to-be-resolved dependencies. We will see a tremendous increase of those in the nearer future, which will also drive build times. By then I expect that a caching solution will have a sizeable practical impact. I'd rather be prepared for it than having the problem first.

From my personal perspective we can use the time of the project's growing to adulthood to test drive Develocity and its caching features. Thank you for the generous offer, @fgurr. I'll leave it, however, to our project's experts on the matter to decide whether this is a option path we want to go down. @dcalavrezo, can you please chime in on viability or, alternatively, bring this to the infrastructure team's attention and get back with a decision?

@thiloschmitt @fgurr thank you both for your input. You are both spot on with your comments. The S-CORE project it still in its early phases. We're not even compiling any source code per se ( yet!).

Once we'll reach that phase and start adding source code modules and integrate different components, the build time will increase exponentially.

I think that it will be great if you could share the Develocity credentials with us so that we can undergo some caching tests.

Thank you both!

You can connect with your eclipse account to the develocity platform: https://develocity-staging.eclipse.org

I have set up a token for CI purpose: https://github.com/eclipse-score/.eclipsefdn/pull/34

Documentation can be found here: https://eclipse-cbi.github.io/cbi-website/develocity/readme/index.html

@heurtemattes quick question: the documentation only mentions maven/gradle, but this doesn't mean that the instance doesn't support bazel, correct?

Bazel has never been tested with the instance, but it is supported by develocity.

From the official documentation: https://docs.gradle.com/develocity/bazel-configuration/

@heurtemattes my eclipse credentials don't seem to work

The reason is that you're not a committer; you're a contributor to the Score project. Maybe this is something we need to consider.

I'll get back to you next week.

assigned to @fgurr