Use git CLI instead of cURL to fetch spec data
The current POC that is live uses cURL to retrieve specs at build time to compile the documentation for said specs. While this could potentially continue to work, I believe that using Git would be more sustainable for future usage in builds. In the past, we have had issues where applications (namely Github in this case) would have an aggressively set unauthenticated request limit for raw content to prevent abuse. As our clusters share IPs for the build containers, this led to a lot of projects not being able to request data and breaking down the line.
To accomplish this in the most efficient way, we will need an update to Git as the sparse-checkout feature was added in 2020, and our current Git version on Jenkins is from late 2018 (v2.20.1). @fgurr @mward is there a way to update this version to provide more recent functionality and updates, or is the Git version locked by our Jenkins instance?
While we could do this without sparse checkouts, it makes it a lot lighter on network traffic as it only checks out the root files + the spec file, meaning we can ignore any number of build files, resources, or large binaries that might be present in projects. While it could be called pre-optimization, the build already takes ~5 minutes with maybe a third of our current specs. Any optimizations in the process are probably a good thing at this point.