[Data Ingestion and Storage] - Streaming API - Add new source data for Object Storage Connectors

Add a new source data for an Object Storage Source like DBs with the connection information required in order to retrieve the object/objects.

Here we can see the different attributes for the following Object Storage engines: AWS S3, MinIO, Azure Blob Storage. There are common attributes but there are some differences between the required attributes of all of them.

Let's start with MinIO:

Parameter	Required	Description
endpoint_url	✅ Required	Custom MinIO endpoint (e.g., http://localhost:9000)
access_key	✅ Required	MinIO access key
secret_key	✅ Required	MinIO secret key
bucket_name	✅ Required	Name of the MinIO bucket
use_ssl	✅ Required	true for HTTPS, false for HTTP
filter	❌ Optional	Filter objects based on prefix, suffix, date range, etc.

An example of filter could be:

 "filter": {
      "prefix": "logs/",
      "suffix": ".csv",
      },
  "field_delimiter": ";"
  "decimal_delimiter": ".",
  "encoding": "uft8" # use standard encoding codes
  }