Draft: Dev ptq

TODO

• Normalize layers
  • Get weight max value for each layer (Fc/Conv)-> alpha
  • Divide weights by alpha
  • Divide bias by alpha
  • Add PerOutputChannel normalization
    • TODO : add steps to do this ...
• Normalize activation
  • Normalize stimuli between [0;1]
    • or [-1,1]
  • Forward on validation dataset
  • Get max value of activation for each layer -> beta
    • Develop hook sytsem
    • Develop get max activation hook
  • Add scaling factor beta to the activation
  • Add scaling factor beta to the bias
  • Rescale activation by parent bias scaling
• Quantization
  • Input: multiply by (2^n-1)-1 (singed) or (2^n-1) (unsigned)
  • Weights: Multiply by (2^n-1)-1 + round + store as a signed integer of nbbits size
  • Biases: Multiply by (2^n-1)-1 + Multiply by (2^n-1)-1 (singed) or (2^n-1) (unsigned) + store as a signed integer of nbbits size
  • Activation scaling:
    • Input scaling: (2^n-1)-1 + Multiply by (2^n-1)-1 (singed) or (2^n-1) (unsigned)
    • Output scaling: (2^n-1)-1 (singed) or (2^n-1) (unsigned)
    • Activation sclaing : divide by (input scaling divided bu output_scaling)
• Activaiton clipping
  • generate histogram of outputs for each layer (using validation dataset)
  • MSE
    • TODO : add steps to do this
  • KL-Divergence
    • TODO : add steps to do this
• Weights clipping
  • TODO : add steps to do this
• Better scaling operation
  • Fixed-point scaling
  • Single shift scaling
  • Double shift scaling
• Bind method in Python
• LeNet integration test
• Refactor OutputRange hook to use Tensor Getter/Setter
• Add docstring to OutputRange hook
• Refactor hook system to not use registrar ?

aidge_quantization/unit_tests/test_ptq.py

@@ -2,12 +2,9 @@
 import aidge_core
 import aidge_backend_cpu
 import aidge_onnx
+import aidge_quantization
 import numpy as np
 import matplotlib.pyplot as plt
-from jinja2 import Environment, FileSystemLoader
-import os
-import pathlib
-import onnx
 
 model = aidge_onnx.load_onnx("MNIST_model/LeNet_MNIST.onnx")
 
@@ -29,18 +26,7 @@ input_node.get_operator().set_backend("cpu")
 ## Link Producer to the Graph
 input_node.add_child(model)
 
-nodesRegexes = {}
-nodesRegexes["Flatten"] = aidge_core.NodeRegex("Flatten")
-
-# Graph Regex
-graphRegex = ["Flatten;"]
-
-graphMatching = aidge_core.GRegex(nodesRegexes, graphRegex)
-all_match = graphMatching.match(model)
-print('Number of match : ', all_match.get_nb_match())
-
-for mn in all_match.get_match_nodes():
-    aidge_core.remove_flatten(mn)
+aidge_core.remove_flatten(model)
 
 model.save("my_supported_LeNet")
 
@@ -73,4 +59,4 @@ for outNode in model.get_output_nodes():
 
 #add PTQ part for testing
 #created ordered graph, as in cpp and do
-#quantizeNetwork(ordered_graph_view, 8, verbose);
\ No newline at end of file
+aidge_quantization.quantizeNetwork(scheduler.get_static_scheduling(), 8, True)