Skip to content

Add property-based tests to exporter

Context

Current tests are mostly written by hand with some expertise required to identify corner cases and assess where failure is expected, especially regarding broadcasting of dimensions and values over dimensions. While a number of issues have been identified that way, it is both time consuming, error prone, and creates numerous test cases.

The shift to property-based testing would allow the automated generation of test cases, their evaluation, and the memoization of failures where required, with minimal user input. It does however require the definition of strategies to generate the tested components of the framework, and the definition of properties to validate for each.

Property-based testing is supported by the Hypothesis Python framework.

Modified files

  • Supporting property-based testing

    • test/strategies/*.py, to add strategies to generate various operators (nodes.py), and tensors (core.py)
    • test/test_strategies.py, to add test for strategies ensuring the generated elements are valid under AIDGE rules
    • test/operators/*.py, to add property-based tests for specific operators (Add, FC, MatMul, Conv2D)
    • test/export_acetone*.py, removed when replaced by property-based tests
    • test/utils.py, to generalise tests comparing the AIDGE CPU backend and exporters
    • test_export_acas.py, to use new test utils
  • Supporting named tensors and better templates:

    • acetone/generator.py and operators.py and export_registry.py to add support for named tensors in code templates (matching the SONNX specification where applicable (and applied))
    • conv2d.jinja and matmul_forward.jinja to use named tensors
    • conv2d.jinja and matmul_forward.jinja and dims.jinja to use multi-dimensional array and simplify index computation (where applicable)
  • Miscellaneous project management

    • Module __init__.py. to cleanup extraneous imports
    • pyproject.toml and version.txt, to bump dependencies and module version, and fix project description issues
    • .gitgnore, to exclude hypothesis data from version control
    • Removed unused files leftover from the AIDGE module template `MakeLists.txt`
    • Misc fixes for typing and linter issues

Detailed major modifications

  • Added property-based testing support for (most) supported nodes
  • Added support for named tensors in templates
  • Aligned (some) template with the proposed SONNX specification

To be improved

  • Some operators still rely on the AIDGE attributes directory (e.g. Add)
  • Find better strategies to generate values in tensors
  • Uniform strategy for weights and bias generation

Merge request reports

Loading