Skip to content

feat: merge into single Spark job and implement Object Storage results passing, Refs #24, #25

Aristotelis Katsanas requested to merge mergeJobsAndMINIO into develop

Eliminates cold start overhead by combining evaluator_kpi_library_data_payload.py and evaluator_user_defined_rules_generator.py into merged_script.py.

  • Remove HTTP middleware call between jobs
  • Pass data in-memory instead of filesystem I/O
  • Final results are passed through dedicated minIO bucket
  • Add new /evaluate-data-merged endpoint for gradual migration
  • Increase resource allocation (6g/4cores) for combined workload
  • Maintain backward compatibility with existing endpoints

BREAKING CHANGE: None (old endpoints preserved for grace period)

Merge request reports

Loading