Speed up CI fetch job

We are spending around 8 minutes in the fetch job. This really should not be needed.

Investigate why this is so slow. It might be related to the fact that we are not caching downloads, so only doing possibly unpacking huge git repository tarballs multiple times.