Skip to content

Added initial revalidation service to revalidate SDs against schemas

Cristina Pauna requested to merge feature/sdRevalidation into main

Implements the core of #31 (closed).

The re-validation service can be a long-running service that is set up once and occasionally triggered to start the re-validation process. It stars one management Thread and a configurable number of worker Threads. The management Thread will fetch batches of hashes from the Self-Description store and queue them for re-validation. The batch size is configurable. When the number of queued hashes drops below half a batch-size, the manager will fetch a new batch.

Batches are fetched using the last hash of the previous batch, for efficient pagination.

Writing to the database only happens when a Self-Description fails to validate.

The self-descriptions in the store are chunked into a configurable number of chunks, with each hash being assigned to a chunk based on:

hashtext(sdhash) % :chunks = :chunkid

This allows a cluster of catalogue instances to all work on the re-validation process.

Merge request reports

Loading