Skip to content

[Quality Evaluator]issue with the dataset "meteoAggregate_airTemperatureMin" in ITI infrastructure

The following request (executed on localhost with port forwarding to the ITI VM on which Quality Evaluator is running) fail with the following error: "FileNotFoundError: [Errno 2] No such file or directory: '/app spark_jobs/data/meteoAggregate_airTemperatureMin_DQV_quality_profile.json'

To run the following curl, you need to connect to the VM proxy and forward port 8001 where Quality Evaluator is now listening.

curl --location 'http://localhost:8001/evaluate-data' --header 'content-type: application/json' --data '{
    "dataset_id": "meteoAggregate_airTemperatureMin",
    "column_name": "airTemperatureMin",
    "values": [
        null,
        25.36,
        16.65,
        26.18,
        22.5,
        21.93,
        17.66,
        22.53,
        14.4,
        20.92,
        26.1,
        18.91,
        14.68,
        null,
        null,
        16.84,
        24.35,
        15.35,
        21.68,
        17.75
    ]
}'

Quality Evaluator then generates the following error:

[2024-10-29 06:53:39,698] ERROR in app: Exception on /evaluate-data [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/evaluator_middleware.py", line 297, in evaluate_data
    profileResult = waitForDriverKPIwData(spark_master_ip, subID, d_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/evaluator_middleware.py", line 171, in waitForDriverKPIwData
    with open(f'/app/spark_jobs/data/{datId}_DQV_quality_profile.json', 'r') as file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/spark_jobs/data/meteoAggregate_airTemperatureMin_DQV_quality_profile.json'
[2024-10-29 07:01:14,656] ERROR in app: Exception on /evaluate-data [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/evaluator_middleware.py", line 297, in evaluate_data
    profileResult = waitForDriverKPIwData(spark_master_ip, subID, d_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/evaluator_middleware.py", line 171, in waitForDriverKPIwData
    with open(f'/app/spark_jobs/data/{datId}_DQV_quality_profile.json', 'r') as file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/spark_jobs/data/meteoAggregate_airTemperatureMin_DQV_quality_profile.json'
Edited by Antoni Gimeno