Extend metrics definition for automatic ranking of execution runs
Currently it is not possible to implement a general, automatic ranking method for the metrics. This could be achieved by extending the metrics definitions with hints about the comparison-type like this: `
"metrics": {
"type": "regression-metrics/v1",
"date_time": "2023-11-22 15:36:44",
"status_text": "success"
"more_is_better": {
"r_squared": 0.831178119365146,
"adjusted_r_squared": 0.8270170166734419,
},
"less_is_better": {
"mse": 0.0025680503998073007,
"rmse": 0.05067593511527242,
}
}
`
Since we have several metric values, the automatic comparator would count which execution has more "wins" to determine the overall winner. It means that all values have the same weight.
Additionally, I think we need in the system-info also the start and finish time to compute the duration and give context when the execution took place: `
"system_info": {
"system_name": "node03",
"fqdn": "www.ai-lab.nrw",
"cpu": "10",
"gpu": "",
"memory": "65850740Ki",
"start_time": "2023-11-23 16:34:08",
"finish_time": "2023-11-23 16:35:27"
}
`
Edited by Martin Welss