Integration of the Sigmoid function to the PTQ pipeline

Description

Currently, for a network to be quantized using the PTQ algorithm, all of its nodes must have a linear reponse (FC, Conv2D, ..).

The reason behind this requirement is purely mathematical : during the PTQ the input values of each node are calibrated using scalar multiplications by scaling factors.

As a result, the output of a node, which has its inputs rescaled by a scaling factor, has to be equal to the scaling factor multiplied by the output that the node would have if its inputs were untouched. That is :

Node(scaling_factor * x) = scaling_factor * Node(x) (1)

Why is the ReLU an exception ?

The linearity condition is not verified by the ReLU, yet it is seamlessly integrated to the PTQ pipleine. The reason why is also mathematical : the scaling factor is not any real, but only a positive one. As result the output of Node(scaling_factor * x) cannot change sign, and the possible solution to (1) are node responses that are both linear on R+ and on R-, that is ReLUs and LeakyReLUs (and that is really all).

How to integrate a non-linear operator ?

Assume a non-linear, let say a Sigmoid, operator receives a rescaled input. As (1) is not verified, we need to rescale its input back to its original range, by cancelling the product of all the previous scaling factors, and revert this rescaling after the output computation.

Goals of this Merge Request ?

The goal of this MR is not only of integrating the Sigmoid operator, but also to provide a reference example for the integration of any non-linear, non-quantized, operator to the PTQ pipeline.

TODO

• modify insertScalingNodes() to add an extra scaling before non-linear nodes
• modify normalizeParameters() and normalizeActivations() to rescale inputs/outputs of non-linear nodes
• modify quantizeNormalizedNetwork() to keep the float dynamic of non-linear nodes untouched
• Test and validate the changes on several topologies

Note : in addition to theses changes, several modifications were made to improve the code quality.

include/aidge/quantization/PTQ/PTQ.hpp

@@ -41,6 +41,11 @@ namespace Aidge {
      */
     static const std::set<std::string> mergingNodeTypes({"Add", "Concat", "Sub"});
 
+    /**
+     * @brief Set of the types of the nodes that won't be quanized
+     */
+    static const std::set<std::string> notQuantizedNodeTypes({"Sigmoid", "Tanh"});
+
     /**
      * @brief Determine if a node contains an affine transform (that is Y = A.X + B)
      * @param node The node to be checked
@@ -62,6 +67,13 @@ namespace Aidge {
      */
     bool isMerging(std::shared_ptr<Node> node);
 
+    /**
+     * @brief Determine if a node contains an operator that won't be quantized
+     * @param node The node to be checked
+     * @return True if the node is not quantized, else false.
+     */
+    bool isNotQuantized(std::shared_ptr<Node> node);
+
     /**
      * @brief Retrieve the scheduled vector of node of a graphView, without the Producer nodes.
      * @param graphView The graphView containing the nodes