[Add] benchmark mechanism
What commit version of aidge do you use
-
aidge bundle
: 0.5.1_dev
Issue Summary
Aidge currently lacks a standardized and easy-to-use benchmarking mechanism for evaluating the performance and correctness of different modules. To ensure efficient model execution and accurate inference across various backends, we need a flexible benchmarking script.
Motivation
Aidge provides multiple module options for running deep learning models. A proper benchmarking framework will:
- Help compare inference time across different execution backends.
- Verify that inference outputs are consistent across different configurations.
- Assist in optimizing implementations by identifying performance bottlenecks.
Expected Features
The benchmarking script should:
-
Measure Inference Time -
Compare inference results between the assessed modules and ONNXRuntime. - Support testing
-
individual ONNX operator -
custom input size -
custom attributes
-
-
Aidge operators -
custom input size -
custom attributes
-
-
multiple-nodes ONNX model
-
- Include configurable options for
-
number of warmup and test runs -
set of tests or individual model
-
-
Support JSON output for use of benchmark results in other scripts. -
Allow users to specify the library to assess (Aidge module, PyTorch, ONNXRuntime). - Be usable for
-
Aidge unit-tests -
as independant script
-
Edited by Maxence Naud