Skip to content

[Data Discovery] Trino integration in the Discovery Flow (DB Query Snapshot)

To integrate the current CERTH API with the Trino discovery flow, this issue outlines the sequence of API calls.

First analysis in the Miro Board:

Whole sequence (FrontEnd)

1 - DB Query selection (new button)

2 - Set DB connection: a) Create connection: fill in the connection information based on the target DB selected (PostgreSQL, Cassandra, etc.) b) Select existing connection (previously configured)

3 - Trino integration a) Create new query (CERTH API) b) Test query (CERTH API)

c) Save query (CERTH API) Stored in MariaDB through the CERTH API.

d) Run Query (integration with Governance and Storage)

Regarding the 'Run Query' feature, the table below outlines the sequence of API calls used to create the artifact in Governance, store the file generated by query execution, and trigger the pipeline.

Action API / Swagger UI doc Example
Add artifacts Pending First action because the artifactID is needed in the next requests. Depending on the issue: Implement Lineage Information for DB Queries
Get the signed URL for the artifact (StorageAPI) API: StorageAPI, Swagger: http://91.235.109.231:8089/swagger-ui/index.html#/Storage/getArtifactUploadLocation GET /storage/artifacts/{artifactId}/files/{fileName}/upload-location
Put the CSV in MinIO using the signed URL API: MinIO API, Put using the previous signed URL http://91.235.109.231:9000/default/a0cc8c00-7a42-40a9-88f8-5228eaeaad5b/test-file-name?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250227T072421Z&X-Amz-SignedHeaders=host&X-Amz-Credential=minioadmin%2F20250227%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Expires=600&X-Amz-Signature=4ff78d788a786b0b5571459f17b5f9193b0be78646825b17da421d7b38172f17
Register the pipeline in MageAI (Bulk type) API: StreamingAPI, Swagger: http://91.235.109.231:8000/docs#/streaming/create_streaming_streamings__post Example
Start the pipeline API: StreamingAPI, Swagger: http://91.235.109.231:8000/docs#/streaming/start_streaming_streamings__streaming_id__start_put Example: http://91.235.109.231:8000/streamings/93137583-88e9-4800-81bd-0c72fdde9397/start
Edited by Antoni Gimeno