Mage AI - DB Pipeline integration with Quality Evaluator

Context:

• Profiling multiple tables from SQL and NoSQL databases.
• Each column will be processed by the Quality Evaluator service.

Approach:

• Query all tables and export the data to files (CSV format for the first iteration; consider using Parquet files later for improved parallel processing).
• Invoke the Quality Evaluator service for each processed file.
• Publish the aggregated information, including metadata and quality measurements, to the Governance API.