Metrics not being shown in execution-run.json after training.
A metric calculation step is being started directly after the training. Training might take longer but in this example a small dataset is being used so the training is as fast as a inference. Although model has logger.info('MericsAvailable') and logging metrics as shown in the screenshot below:
Here is what is being logged:
Logged metric
2023-12-12 08:57:26,409 __main__ INFO {'metrics': {'type': 'coco-metrics/v1', 'date_time': '2023-12-12 08:57:22', 'status_text': 'success', 'more_is_better': {'AP': 0.0, 'AP_50': 0.0}, 'less_is_better': {}}}
And here is how the execution-run.json after the training:
execution-run.json
{
"probeIndicator": [
{
"value": "false"
}
],
"nodes": [
{
"proto_uri": "org/acumos/d4e836ce-d145-4cab-a2e7-8c0b6bd10ebb/detrmodel/1.0.0/detrmodel-1.0.0.proto",
"image": "cicd.ai4eu-dev.eu:7444/tutorials/detr_objectdetection/detr_model:latest",
"node_type": "MLModel",
"container_name": "detrmodel1",
"operation_signature_list": [
{
"connected_to": [],
"operation_signature": {
"operation_name": "startTraining",
"output_message_name": "TrainStatus",
"input_message_name": "TrainingConfig",
"output_message_stream": false,
"input_message_stream": false
}
}
],
"checksum": ""
},
{
"proto_uri": "org/acumos/7ea21f2e-0631-474f-a9d1-3d4e6ab6839f/detrs3databroker/1.0.0/detrs3databroker-1.0.0.proto",
"image": "cicd.ai4eu-dev.eu:7444/tutorials/detr_objectdetection/detr_s3_databroker:latest",
"node_type": "MLModel",
"container_name": "detrs3databroker1",
"operation_signature_list": [
{
"connected_to": [
{
"container_name": "detrmodel1",
"operation_signature": {
"operation_name": "startTraining"
}
}
],
"operation_signature": {
"operation_name": "send_training_signal",
"output_message_name": "TrainingConfig",
"input_message_name": "Empty",
"output_message_stream": false,
"input_message_stream": false
}
}
],
"checksum": "docker-pullable://cicd.ai4eu-dev.eu:7444/tutorials/detr_objectdetection/detr_s3_databroker@sha256:78d65e6b5ecb4850845b5c34943ea15577931d618dc2b94144465a640e4bedf6"
},
{
"proto_uri": "org/acumos/37dd144b-2d43-49cd-b1a4-bed93541ce6f/detrtensorboard1/1.0.0/detrtensorboard1-1.0.0.proto",
"image": "cicd.ai4eu-dev.eu:7444/tutorials/detr_objectdetection/detr_tensorboard:latest",
"node_type": "MLModel",
"container_name": "detrtensorboard11",
"operation_signature_list": [],
"checksum": "docker-pullable://cicd.ai4eu-dev.eu:7444/tutorials/detr_objectdetection/detr_tensorboard@sha256:d5140ff4a3cc35c24ecc63d716b9c1b9f13770b5cc1c7f2553c8f6284d09aaed"
}
],
"name": "trainingdetr",
"version": "1",
"input_ports": [],
"system_info": {
"system_name": "node03",
"fqdn": "202.61.251.28",
"cpu": "10",
"gpu": "",
"memory": "65850664Ki"
}
}
