Mage AI - Consolidate All Discovery Connector Pipelines into a Single Unified Pipeline

The goal is to merge all Discovery Connectors pipelines in a generic multi-purpose data sources.

The pipeline will be launched via API and will received a payload with the following information. This information may change when the pipeline will be triggered by the API for pipeline management.

Example using the Mage AI API:

{
    "pipeline_run": {
        "variables": {
            "dataset-id": "58c45b18-9889-43d3-8583-13bdcf91499f",
            "connection-info": {
                "type": "cassandra / postgresql / ...",
                "connection-data": {
                   //open json data with the connection info for the different types 
                }
            }
        }
    }
}