[Quality Evaluator]issue with the dataset "meteoAggregate_airTemperatureMin" in ITI infrastructure
The following request (executed on localhost with port forwarding to the ITI VM on which Quality Evaluator is running) fail with the following error: "FileNotFoundError: [Errno 2] No such file or directory: '/app spark_jobs/data/meteoAggregate_airTemperatureMin_DQV_quality_profile.json'
To run the following curl, you need to connect to the VM proxy and forward port 8001 where Quality Evaluator is now listening.
curl --location 'http://localhost:8001/evaluate-data' --header 'content-type: application/json' --data '{
"dataset_id": "meteoAggregate_airTemperatureMin",
"column_name": "airTemperatureMin",
"values": [
null,
25.36,
16.65,
26.18,
22.5,
21.93,
17.66,
22.53,
14.4,
20.92,
26.1,
18.91,
14.68,
null,
null,
16.84,
24.35,
15.35,
21.68,
17.75
]
}'
Quality Evaluator then generates the following error:
[2024-10-29 06:53:39,698] ERROR in app: Exception on /evaluate-data [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/evaluator_middleware.py", line 297, in evaluate_data
profileResult = waitForDriverKPIwData(spark_master_ip, subID, d_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/evaluator_middleware.py", line 171, in waitForDriverKPIwData
with open(f'/app/spark_jobs/data/{datId}_DQV_quality_profile.json', 'r') as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/spark_jobs/data/meteoAggregate_airTemperatureMin_DQV_quality_profile.json'
[2024-10-29 07:01:14,656] ERROR in app: Exception on /evaluate-data [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/evaluator_middleware.py", line 297, in evaluate_data
profileResult = waitForDriverKPIwData(spark_master_ip, subID, d_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/evaluator_middleware.py", line 171, in waitForDriverKPIwData
with open(f'/app/spark_jobs/data/{datId}_DQV_quality_profile.json', 'r') as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/spark_jobs/data/meteoAggregate_airTemperatureMin_DQV_quality_profile.json'
Edited by Antoni Gimeno