Skip to content

CI: Increase server timeout and separate disk cache

Stephen Ryan requested to merge sr/ci_increase_timeout into main

The GC is not cleaning the disk cache, because the bazel server does not persist between builds. See Issue #24.

The disk cache and output base were separated. After some experimenting, I found that the GC cleanup of the disk cache can lead to problems when the output base is in the same directory as the disk cache.

The GC can remove extracted bazel .jar files from install directory, which will cause all bazel commands, including shutdown, to fail. This can be resolved by deleting the directory so that it can be re-extracted by bazelisk. Another problem is that some important files in the output base for the current directory could be deleted, and all bazel commands will fail, for example, because some BUILD file is missing somewhere in the output base. Each MR runs in a different directory, and creates a new bazel output base. The files in these directories will be removed by the GC, but the directories and dangling symlinks will stay, so they will have to be cleaned eventually.

Because of these issues, I think we should keep them separated. It would be good to have a job that removes the output base when a PR is merged, abandoned, or is older than a specified age.

  • Use environment variable to keep the bazel server running.
  • Make new directories for the disk cache and output base
  • Prepend --output_user_root="${OUTPUT_USER_ROOT}" to all bazel commands to use the new output base
  • By default, the server timeout is 3 hours because we are no longer using the TEST_TMPDIR environment variable. But --max_idle_secs=10800is set to be sure.
Edited by Stephen Ryan

Merge request reports

Loading