Setup a Yocto sources mirror to alleviate long download times

Description

Building oniro-image-base on supported Oniro targets takes a very long time for the first time due to long download times for component sources from the various source forges.

Besides, it is a good idea to have a mirror of white-listed sources that Oniro depends on from a supply-chain availability point of view.

There are two possible approaches:

1. A transparent proxy setup similar to squid-deb-proxy-server that mirrors any package that is requested by users building Oniro-based projects.
2. A CI-based job that updates the mirror with any missing files, possibly based on some whitelist rules that IP compliance team could help put together.

The second approach is probably simpler to setup and has less liability issues in case of mirroring files.

changed milestone to %goofy alpha

added teaminfrastructure label

set weight to 4

added to epic &125

changed iteration to Oniro Core Iterations Apr 1, 2022 - Apr 30, 2022

@alpianon @cpiana Any guidelines on how to approach this from a legal PoV?

changed title from Setup a CI-based component mirror for components in oniro-image-base to alleviate long download times to Setup a CI-based component mirror to alleviate long download times

assigned to @pastanki

changed title from Setup a CI-based component mirror to alleviate long download times to Setup a Yocto sources mirror to alleviate long download times

@idlethread using the debian (|fedora, in the future) matching criterion would be a good starting point.

changed the description

added Doing label

changed iteration to Oniro Core Iterations May 1, 2022 - May 31, 2022

added 1d 6h of time spent

When you get to this, we should consider if it is possible to migrate our current mirror setup in China for meta-openharmony to the new setup. Please contact me regarding this when it becomes relevant.

changed iteration to Oniro Core Iterations Jun 1, 2022 - Jun 30, 2022

changed epic to &126

changed milestone to %goofy beta

changed iteration to Oniro Core Iterations Jul 1, 2022 - Jul 31, 2022

1. Tested:

• squid-deb-proxy - it is running for several forks and I'm still gathering what deb sources need to be whitelisted
• squid for CI jobs - added as http_proxy and https_proxy CI environmental variables/ gitlab-runner variables.

The proxy is set both for WRC and OTC.

added 2d of time spent

set weight to 5

changed iteration to Oniro Core Iterations Aug 1, 2022 - Aug 31, 2022

Related to issue

added 2d of time spent

changed iteration to Oniro Core Iterations Sep 1, 2022 - Sep 30, 2022