[add] Element wise backward

Context

Fix #37 (closed)

Modified Files

• MulImpl.hpp and MulImpl_kernels.hpp : Refactor backward pass of Mul, see explantions below.
• MulImpl.cpp
• Test_MulImpl.cpp : Test the new backward pass. The tests consists of 4 tests with fixed values, and one with random values (but fixed dimensions).

Description of the changes

Overview

The backward kernel computes gradients for the multiplication operation while handling broadcasting. For each element in the output gradient tensor, it:

1. Determines which input elements contributed to this output (considering broadcasting)
2. Applies the chain rule for multiplication to compute gradients

    grad_input_0[i] += grad_output[k] * input1[j]
    grad_input_1[j] += grad_output[k] * input0[i]
    where i,j are the contributing indices from input0 and input1 for output position k

Example

`input0` [1,3]: [[1,2,3]]
`input1` [4,3]: [[4,5,6],
                 [7,8,9],
                 [1,2,3],
                 [4,5,6]]

Output shape: [4,3], thus `outputDims` = [4,3], `dims0` = [1,3], and `dims1` = [4,3].
Broadcasted shapes:
- getBroadcastedDims(outputDims, dims0): [1,3]
- getBroadcastedDims(outputDims, dims1): [4,3]

For instance, assume we multiply two tensors A and B, with A of shape [1,3] and B of shape [4,3].

input0 = [[1,2,3]]

input1 = [[4, 5, 6], [7, 8, 9], [1, 2, 3], [4, 5, 6]]

Let's also assumes that the output gradient is full of 1s.

Then we iterate for i = 0 to output.size():

• At iteration 0 / index [0,0]

  → The element at index [0,0] of input0 contributed to the output value

  → Thelement at index [0,0] of input1 contributed to the output value.

grad_input_0[0] += grad_output[0,0] * input0[0,0]
grad_input_1[0] += grad_output[0,0] * input1[0,0]

• At iteration 1 / index [0,1]

  → The element at index [0,1] of input0 contributed to the output value.

  → The element at index [0,1] of input1 contributed to the output value.

grad_input_0[1] += grad_output[0,1] * input1[0,1]
grad_input_1[1] += grad_output[0,1] * input0[0,1]

• At Iteration 2 / index [0,2]

  → The element at index [0,2] of input0 contributed to the output value.

  → The element at index [0,2] of input1 contributed to the output value.

grad_input_0[2] += grad_output[0,2] * input1[0,2] 
grad_input_1[2] += grad_output[0,2] * input0[0,2]

• At Iteration 3 / index [1,0]

  → The element at index [0,0] of input0 contributed to the output value (due to broadcasting!). This can be determined by looking at the dimensions of getBroadcastedDims(outputDims, dims0), which is 1 for index 0. It indicates that the dimension has been broacasted for the output, or that both dimensions are equal to 1.

  → The element at index [1,0] of input1 contributed to the output value.

// Updates:
grad_input_0[0] += grad_output[1,0] * input1[1,0]
grad_input_1[3] += grad_output[1,0] * input0[0,0]

etc.

changed the description

added 11 commits

• 8d707eff - 1 earlier commit
• 279f405c - Hotfix for more generic registry
• 5e17399e - Update backend_cpu with https://gitlab.eclipse.org/eclipse/aidge/aidge_core/-/merge_requests/277
• 7593372e - Merge branch 'RemoveSCM' into 'dev'
• 41205e8c - WIP: Improved Mul Backward pass
• 7e95d762 - WIP/TODO Test Mul bacward function
• 4e63f48b - WIP Improve mul backward kernel
• b77b76f8 - chore: Rename variables and add a new test section
• a6623bdc - chore: Refactor Mul operator test
• f5cf8bf6 - chore: Comply with clang-tidy rules
• 8af148a8 - chore: Rename test sections

Compare with previous version

marked this merge request as ready

changed target branch from main to dev

assigned to @jeromeh

Is this really ready to be merged? I see commented code with some TODO

added 1 commit

• 03c26cc9 - chore: Remove commented code

Compare with previous version

My bad, should be fixed in commit 03c26cc9.

added 16 commits

• 03c26cc9...03e4e4b1 - 8 commits from branch eclipse/aidge:dev
• a12b19e7 - WIP: Improved Mul Backward pass
• ee7959c9 - WIP/TODO Test Mul bacward function
• 7655f8c2 - WIP Improve mul backward kernel
• f9503df7 - chore: Rename variables and add a new test section
• b79dfa7a - chore: Refactor Mul operator test
• 48217370 - chore: Comply with clang-tidy rules
• 487d33d3 - chore: Rename test sections
• 7dfa7a41 - chore: Remove commented code

Compare with previous version

added Feature 🚀 StatusReview Ready TopicOperator labels

changed title from Element wise backward to [add] Element wise backward

changed milestone to %aidge v0.5.0

requested review from @cmoineau

requested review from @pineapple

Adding a test for backpropagation is a good thing. Is it what you did?

This way of computing gradient is still far from being efficient. What have you changed?

Since you also change the format, it is hard to see what you really changed and so doing any comment

added 1 commit

• d0b0ef0a - chore: Rename test section to be more explicit

Compare with previous version

added 1 commit

• 104de16d - chore: Rename a test section

Compare with previous version

changed the description

added 1 commit

• 7361d007 - chore: Rename some variables to be more explicit

Compare with previous version

added 26 commits

• 7361d007...07996a12 - 15 commits from branch eclipse/aidge:dev
• ab18b9db - 1 earlier commit
• 36a15c19 - WIP/TODO Test Mul bacward function
• 76888f9a - WIP Improve mul backward kernel
• adc30d44 - chore: Rename variables and add a new test section
• 4d681856 - chore: Refactor Mul operator test
• 49b0a2c1 - chore: Comply with clang-tidy rules
• 191881f5 - chore: Rename test sections
• 26635709 - chore: Remove commented code
• 8f237f4c - chore: Rename test section to be more explicit
• 6acc82fe - chore: Rename a test section
• 1c5ca468 - chore: Rename some variables to be more explicit

Compare with previous version

resolved all threads

approved this merge request

changed the description

added 14 commits

• 1c5ca468...7d527920 - 8 commits from branch eclipse/aidge:dev
• 11103c91 - WIP: Improved Mul Backward pass
• 8a23e6fb - WIP/TODO Test Mul bacward function
• a4b0e02a - WIP Improve mul backward kernel
• b6f651b9 - chore: Rename variables and add a new test section
• 1101341c - chore: Remove commented code
• 63ccf533 - chore: Rename some variables to be more explicit

Compare with previous version

reset approvals from @pineapple by pushing to the branch

changed the description

added 3 commits

• dc6c7efd - Change 'weightInterLeaving' for 'weightInterLeaved'
• fc8967c7 - UPD: change log::info of some tests to the new format
• 5a1b833d - chore: Improve and test Mul Backward kernel

Compare with previous version

added 10 commits

• 5a1b833d...7d527920 - 4 commits from branch eclipse/aidge:dev
• 11103c91 - WIP: Improved Mul Backward pass
• 8a23e6fb - WIP/TODO Test Mul bacward function
• a4b0e02a - WIP Improve mul backward kernel
• b6f651b9 - chore: Rename variables and add a new test section
• 1101341c - chore: Remove commented code
• 63ccf533 - chore: Rename some variables to be more explicit

Compare with previous version

added 1 commit

• 09a81344 - chore: Improve and test Mul Backward kernel

Compare with previous version

added 1 commit

• 4fbee26f - chore: Improve and test Mul Backward kernel

Compare with previous version

resolved all threads

added 20 commits

• 4fbee26f...7302dd25 - 19 commits from branch eclipse/aidge:dev
• 5de68dcc - chore: Improve and test Mul Backward kernel

Compare with previous version

mentioned in commit 9d427c11

merged

mentioned in issue #30 (closed)

mentioned in merge request !134 (merged)