Speed up CI fetch job
We are spending around 8 minutes in the fetch
job. This really should not be needed.
Investigate why this is so slow. It might be related to the fact that we are not caching downloads, so only doing possibly unpacking huge git repository tarballs multiple times.