AdaptToBackend CostantFolding cannot generate scheduling

added Bug 🐛 Discussion 💬 Help Wanted labels

assigned to @wboussella, @cmoineau, and @olivierbichler

I did not know adapt to backend generated a metaOp, current expand metaOp is IMO an ok solution. Best solution will be to tweak MetaOperator registration to generate ExportNode of the node inside of the metaoperator if the metaorperator don't have an implementation (the same way it behave for backend ...)

I don't understand what you were expecting from constant folding as the weight has already been transposed (just look at the dimensions of your weight tensor).

Furthermore with the message:

RuntimeError: missing or invalid registrar key: [Any, NHWC, [], Int8, NHWC, [(2, 2), (1, 1), (1, 1), (2, 2)], Int32, NHWC, [(2, 2)]], [Float32, Default, []] for registrable object N5Aidge16OperatorImpl_cpuINS_7Conv_OpILh2EEEFvRKSt5arrayImLm2EES6_S6_RKS3_ImLm4EEmPKvSB_SB_PvEFvvEEE

We see that the graph has been well changed as the input are noted NHWC.

Problem seems to arise from the fact that the scaling factor data format is default (NHWC).

Why is scaling factor producer not in the graph?

Problem seems to arise from the fact that the scaling factor data format is default (NHWC). : I don't understand, do you mean the Quantizer ? The only registered ops that i need are Producers and the custom metaOP PaddedConvScalingRelu.

Why is scaling factor producer not in the graph? : I use the code example provided in the issue to create my model. You're talking about scale_MulQuant_ScalingFactor ? (expand quantizer)

%%{init: {'flowchart': { 'curve': 'monotoneY'}, 'fontFamily': 'Verdana' } }%%
flowchart TB

Producer_0(<em>Producer#0</em>):::producerCls_rootCls
Producer_1(<em>Producer#1</em>):::producerCls
Conv2D_0("conv<br/><sub><em>(Conv2D#0)</em></sub>")
Round_0("scale_RoundQuant<br/><sub><em>(Round#0)</em></sub>")
Clip_0("scale_ClipQuant<br/><sub><em>(Clip#0)</em></sub>")
Mul_0("scale_MulQuant<br/><sub><em>(Mul#0)</em></sub>")
Producer_2("scale_MulQuant_ScalingFactor<br/><sub><em>(Producer#2)</em></sub>"):::producerCls
Transpose_0(<em>Transpose#0</em>)
Producer_0-->|"0 [2, 1, 1, 2] Int8<br/>&darr;<br/>1"|Conv2D_0
Producer_1-->|"0 [2] Int32<br/>&darr;<br/>2"|Conv2D_0
Conv2D_0-->|"0<br/>&darr;<br/>0"|Mul_0
Round_0-->|"0<br/>&darr;<br/>0"|Clip_0
Mul_0-->|"0<br/>&darr;<br/>0"|Round_0
Producer_2-->|"0 [1] Float32<br/>&darr;<br/>1"|Mul_0
Transpose_0-->|"0<br/>&darr;<br/>0"|Conv2D_0
input0((in#0)):::inputCls--->|"&darr;<br/>0"|Transpose_0
input1((in#1)):::inputCls--->|"&darr;<br/>1"|Clip_0
input2((in#2)):::inputCls--->|"&darr;<br/>2"|Clip_0
Clip_0--->|"0<br/>&darr;"|output0((out#0)):::outputCls
classDef inputCls fill:#afa
classDef outputCls fill:#ffa
classDef externalCls fill:#ccc
classDef producerCls fill:#ccf
classDef genericCls fill:#f9f9ff,stroke-width:1px,stroke-dasharray: 5 5
classDef metaCls stroke-width:5px
classDef rootCls stroke:#f00
classDef producerCls_rootCls stroke:#f00,fill:#ccf
classDef genericCls_rootCls stroke:#f00,fill:#f9f9ff,stroke-width:1px,stroke-dasharray: 5 5
classDef metaCls_rootCls stroke:#f00,stroke-width:5px

What I mean is that the error indicate that one of the tensor is Default format (NCHW) and float32 [Float32, Default, []] but in your registration you forced NHWC, so looks like this is your error.

I thought it was the scaling factor but it looks like it is instead the output of the conv operator.

It seems logic that adapt to backend change only the inputs and not the outputs. It is possible that it is due to the fact that we need to change the forward dims so that the produced tensor is NHWC.

thanks for your answer, I've fix the issue by register producer from the quantizer with aidgeArm backend which have a dtformat.any, like this :


from aidge_export_arm_cortexm.export_registry import ExportLibAidgeARM, ExportLibCMSISNN

# Utilisation du Backend "cmsis_nn" de ExportLibCMSISNN pour les op 
for node in model.get_nodes():
    if node.type() == "PaddedConvScalingRelu":
        print(str(node))
        for n_in_pscr in node.get_operator().get_micro_graph().get_nodes():
            if n_in_pscr.type() == "Quantizer" :
                for n_in_quantizer in node.get_operator().get_micro_graph().get_nodes():
                    if n_in_quantizer.type() == "Producer":
                        n_in_quantizer.get_operator().set_backend(ExportLibAidgeARM._name)
            else :
                if n_in_pscr.type() == "Producer":
                    n_in_pscr.get_operator().attr.constant = True
                n_in_pscr.get_operator().set_backend(ExportLibCMSISNN._name)
    node.get_operator().set_backend(ExportLibCMSISNN._name)



print("bf adapt")
aidge_core.adapt_to_backend(model)
model.save("adapt")

aidge_core.expand_metaops(model)
model.save("expand0")

aidge_core.constant_folding(model)
model.save("constant_folding")

Previously i was asking that : Finally, an open question: how do we go about exporting models that need specific metaOPs, as in the case of cmsis_nn with PaddedConvScalingRelu? Is it up to the user to do this or us during the export?

Wouldn't it be better to fuse and set the backends independently in each export, because in my specific case I don't want to modify the whole scheduler_export.py file which is common to all exports?

@axelfarr do you have a view on this?

Graph transformation need to be specific for each export and not in scheduler export!

Multiple issues are fixed on !290 (merged).

@wboussella Does the latest merge of !290 (merged) effectively solves your issue? If so, I let you close it, or comment additionnal issues.

No feedback provided, I assume that the issue is fixed. Please re-open it if you feel that it is not yet fully addressed.

closed

update : now we can generate scheduling with a Sequential->setBackend->adaptToBackend->generateScheduling. But we cannot use constant_folding(), the problem is that we can't forward with a custom backend bc we don't have an impl. We need to find an hybrid solution using cpu backend's and the custom backend. An unit-test is here to confirm the issue !336 @cmoineau @olivierbichler

New issue opened on this subject #241

reopened

closed

AdaptToBackend CostantFolding cannot generate scheduling

What commit version of aidge do you use

Problem description

Reproducible example code

Designs

Child items ...

Activity