Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
D
data-discovery-connectors
Manage
Activity
Members
Labels
Plan
Issues
2
Issue boards
Milestones
Iterations
Wiki
Requirements
Code
Merge requests
0
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Test cases
Artifacts
Deploy
Releases
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Insights
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Eclipse Research Labs
DATAMITE project
data-support-tools
data-discovery-connectors
Merge requests
!3
psnc connectors
Code
Review changes
Check out branch
Download
Patches
Plain diff
Merged
psnc connectors
rapw3k/data-discovery-connectors:main
into
main
Overview
0
Commits
6
Pipelines
0
Changes
4
Merged
Raul Palma
requested to merge
rapw3k/data-discovery-connectors:main
into
main
10 months ago
Overview
0
Commits
6
Pipelines
0
Changes
4
Expand
psnc connectors
1
0
Merge request reports
Compare
main
main (base)
and
latest version
latest version
d7fe45bf
6 commits,
10 months ago
4 files
+
187
−
0
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
Files
4
Search (e.g. *.vue) (Ctrl+P)
app/connectors/psnc/kafka_conn.py
0 → 100644
+
41
−
0
Options
# Ensure to import only the necessary library for connecting to the storage solution selected
import
subprocess
try
:
from
confluent_kafka.schema_registry.schema_registry_client
import
SchemaRegistryClient
except
ImportError
:
subprocess
.
run
([
"
pip
"
,
"
install
"
,
"
confluent-kafka
"
])
from
confluent_kafka.schema_registry.schema_registry_client
import
SchemaRegistryClient
###
# Connect to Kafka based on config_conn params and return metadata_obj
###
def
db_conn
(
schema_registry_url
):
metadata_obj
=
[]
try
:
print
(
"
-> Trying to connect to Kafka...
"
)
# Initialize Kafka client
schema_registry_client
=
SchemaRegistryClient
({
"
url
"
:
schema_registry_url
[
"
endpoint
"
]})
# Retrieve list of subjects
subjects
=
schema_registry_client
.
get_subjects
()
# Retrieve metadata for each subject
for
subject
in
subjects
:
metadata
=
schema_registry_client
.
get_latest_version
(
subject
)
metadata_info
=
{
"
subject
"
:
subject
,
"
schema
"
:
metadata
.
schema
,
"
schema_id
"
:
metadata
.
schema_id
,
"
version
"
:
metadata
.
version
,
}
metadata_obj
.
append
(
metadata_info
)
print
(
"
-> Metadata from Kafka collected.
"
)
except
Exception
as
error
:
print
(
"
-> Error while connecting to Kafka:
"
,
error
)
return
metadata_obj
Loading