Refactor unit tests

Problems description

• Currently pytest are not ran by CI which means we are skipping more than half the tests every time...
• Test are not unifmormized and some are redundant with one another
• Some test are very light and don't test much
• No unit test on specific function used for import or export.

List of current tests

Here is the list of current tests on ONNX repo (in alphabetic order), with a description of what the tests do and indentified TODO:

• compare_layer.py
  • Not a test script, but a file that add utilitary to compare layer by layer the output of onnxrt and aidge.
  • ⚠️ Deprecated, should be updated ...
• test_converter_register
  • Test that each operators are well registered (only check that key exist)
  • TODO: Add a check that the registered method has the right signature (using inspect lib)
  • TODO: Add a check that all registered operator are tested
• test_generic
  • Try to import an onnx with a custom operator, (no assert are done)
  • Export back the ONNX and assert that the generated ONNX is valid
  • TODO: Use ONNX lib to assert that producers are well converted to initializers
  • TODO: Use ONNX lib to assert that atributes are well converted
  • TODO: Refactor this by avoiding saving ONNX and keeping it at ModelProto level
• test_import_export
  • Import and export a mobilenetv2, check the generated file is valid
  • This test is very lack luster and redundant with test_model_zoo
• test_model_zoo
  • ⚠️ This test does nothing right now as the test case is commented
  • This test use pytest
  • Load a model that is stored online (hugging face)
  • Check that model has native coverage
  • Check that forward_dims does not fail (⚠️ No test that forward dims is valid)
  • Try to export back the model and check model validity.
    • Model validity may be true but model not be valid for onnx runtime. We should add a check that the model can be loaded by onnxrt
    • TODO: opset is set fixed to 18, we should get the opset of the loaded ref model
  • Run a forward pass
    • ⚠️ No test at all are done here
    • TODO: refactor compare_layer script to test the forward (note compare_èlayer script should be a utilitary of aidge_onnx instaed of just a script in unit_tests
• test_models
  • Load a vww model, remove flatten and do a forward
  • Assert that the numer of scheduled operator is equal to 114
  • This test is very lack luster and redundant with test_model_zoo
• test_onnx_models_import
  • This test use pytest
  • Load a graph that is stored locally (no loading from a hugghing face model!)
  • Test that no GenericOperator is created
  • This test is redundant with test_model_zoo
• test_onnx_models_import_forward
  • This test use pytest
  • Load a graph that is stored locally (no loading from a hugghing face model!)
  • Run inference of ONNXRT and Aidge and compare outputs
  • This test is redundant with test_model_zoo
• test_onnx_nodes_import_forward
  • This test use pytest
  • based on test_onnx_nodes_import
  • import node and test The forward based on ONNXRT
• test_onnx_nodes_import
  • This test use pytest
  • Based on ONNX test backend
  • import node and test if it is a GenericOperator
• test_unit_cases_import_forward
  • This test use pytest
• test_unit_cases_import
  • This test use pytest

Question of pytest vs unit test

Having both frameworks can be a little confusing, I propose to discuss which one to use and refactor

Pro unit_test:

• Available in base Python
• Simple class syntax

Pro PyTest

• Need to install pytest
• Fixtures
• Parametrize (doable with unittest subtest but more practical with pytest as one can run pytest -k to rerun a case)
• Cleaner report logs

Decision: Since unittest is still compatible with pytest, we decided to switch to pytest will keeping the current unittest test.

TODO

• Update test_onnx_zoo_model (see comment on the section above)
• Move test_models in test_onnx_zoo_model
• Move test_import_export in test_onnx_zoo_model
• Update test_generic (see comment on the section above), tracked by #77
• Update test_converter_register (see comment on the section above)
• Add test on ONNX export
• Move compare_layer.py and update it to use graph edge name