Deploy Pipeline to Jülich HPC

To the goal is to create a file hpc-solution.zip, that contains all files and scripts to submit a pipeline for execution as sbatch job via Unicore to Jülich HPC with gpu use and Web-UI connections.

Preconditions:

• the user has an account on the Jülich system and a sufficient amount of CPU/GPU-hours available.
• the example-pipeline: to be defined: can be detr-object-detection or an LLM pipeline with GPU usage

Flow

• the user extracts the hpc-solution.zip into his home folder
• the zip contains the blueprint.json and all associated protobuf files
• hpc-solution.zip contains a script (python or bash), e.g. "submit-slurm-job.py" that the user must execute on the commandline
• the script can ask for user credentials
• the script creates the necessary sbatch file(s)
• the script uses unicore to submit the job and connect the web-uis
• finally, the script prints the job-id from slurm and the connection endpoints (grpc + http) of the nodes