This epic aims at benchmarking Aidge exports and optimization methods. ## :robot: Targetted networks | Task | Dataset | Model | CPP fp32 | CPP i8 | ARM fp32 | ARM i8 | |:----:|:-------:|-------|----------|--------|----------|--------| | KeyWord spotting | Google Speech Commands | DS-CNN | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Image classification | ImageNet (224x224) | MobileNet-v1 | :heavy_check_mark: | :heavy_check_mark: | :large_orange_diamond: | :large_orange_diamond: | | Visual wake words | Visual Wake Words Dataset | MobileNet-v1 0.25x | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Object detection | COCO (300x300) | SSD-MobileNet-v1 | :x: | :x: | :x: | :x: | | Object detection | MS-COCO 2017 | SSD-MobileNet-v2 | :x: | :x: | :x: | :x: | | Image classification | CIFAR10 | ResNet-8 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Image classification | ImageNet (224x224) | Resnet50-v1.5 | :heavy_check_mark: | :heavy_check_mark: | :large_orange_diamond: | :large_orange_diamond: | | Anomaly detection | ToyADMOS | Deep AutoEncoder | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Image segmentation | KITS 2019 (602x512x512) | 3D UNET | :x: | :x: | :x: | :x: | | Language processing | SQUAD 1.1 | MobileBert | :x: | :x: | :x: | :x: | :heavy_check_mark: : Supported and an example script is provided :x: : May not be supported :large_orange_diamond: : The model is supported but won't fit into memory So far, for the Arm CortexM export, the following targets have been tested : - stm32h743 Nucleo 144 ## :gear: Hardware targets - ARM CortexM - x86 - ARM CortexA (optional/future) ## :toolbox: Frameworks to compare to - TFLite - ExecuTorch - CubeAI - TVM - ONNXRT For each framework measure inference time, memory and accuracy with a model in Float32 and INT8 (same quantization) ## Method to benchmark - Tensor Decomposition - Quantization - PTQ - INT8 - INT4 - SAT - INT8 - INT4 - LSQ - INT8 - INT4 ## Benchmark <table> <tr> <th>Board</th> <th>Model</th> <th>dtype</th> <th>CubeAI (µs)</th> <th>Arm (µs)</th> <th>TFLITE (µs)</th> </tr> <tr> <td>H743</td> <td>LeNet</td> <td>fp32</td> <td>158 728</td> <td>171 863</td> <td></td> </tr> <tr> <td> \- </td> <td> \- </td> <td>i8</td> <td> :x: </td> <td>144 127</td> <td></td> </tr> <tr> <td> \- </td> <td>DS-CNN</td> <td>fp32</td> <td>225 500</td> <td>194 917</td> <td>482 041</td> </tr> <tr> <td> \- </td> <td> \- </td> <td>i8</td> <td>40 050</td> <td> 150 807 22 852 (cmsis-nn) </td> <td>13 841</td> </tr> <tr> <td> \- </td> <td>Deep Autoencoder</td> <td>fp32</td> <td>16 796</td> <td>22 050</td> <td>33 669</td> </tr> <tr> <td> \- </td> <td> \- </td> <td>i8</td> <td>3 601</td> <td> 14 932 4 993 (cmsis-nn) </td> <td>1 323</td> </tr> <tr> <td> \- </td> <td>MobileNetv1 VWW</td> <td>fp32</td> <td>662 880</td> <td>592 770</td> <td>195 615</td> </tr> <tr> <td> \- </td> <td> \- </td> <td>i8</td> <td>85 237</td> <td> 421 047 80 449 (cmsis-nn) </td> <td>45 472</td> </tr> <tr> <td> \- </td> <td>Resnet8</td> <td>fp32</td> <td>721 558</td> <td>879 604</td> <td>527 008</td> </tr> <tr> <td> \- </td> <td> \- </td> <td>i8</td> <td>144 548</td> <td> 666 590 112 322 (cmsis-nn) </td> <td>56 753</td> </tr> </table>