Add Adam optimizer

Context

Add Adam optimizer which usually provides better results compared to SGD.

Add a unit test for Adam optimizer.

This merge request needs the following merge request to be merged : !142

added Feature 🚀 label

approved this merge request

enabled an automatic merge when the pipeline for 6ce8db59 succeeds

@oantoni There seems to be missing includes.

I agree with the missing includes, but this is not related to this MR. I fixed the missing includes for MSE loss.

added 1 commit

• fe35ed3b - Fix missing includes for MSE loss

Compare with previous version

reset approvals from @olivierbichler by pushing to the branch

added 1 commit

• 54f02a04 - Make clean

Compare with previous version

added 1 commit

• 1040710a - Make clean

Compare with previous version

Pipeline failure was due to missing Tensor::sqrt that has just been merged. Please launch again the integration pipeline to check everything is ok.

An include is missing: you should include

#include "aidge/backend/cpu/operator/SqrtImpl.hpp"

in order to "load" the Sqrt cpu impl.

I assume you are talking about the unit test for Adam optimizer. Sorry I was not able to launch the unit test locally before asking for MR. Missing include was added in the unit test.

Oups, there is more missing includes!

I also added the following includes as for SGD :

#include "aidge/backend/cpu/operator/AddImpl.hpp"
#include "aidge/backend/cpu/operator/MulImpl.hpp"
#include "aidge/backend/cpu/operator/SubImpl.hpp"

Please could you try again ?

You forget Div

Added

There is a precision issue now in the test

For the test, let's code exactly the same way as in Adam optimizer.

Still not good. I suggest you run the test locally to find the problem.

I'm sorry for that. There was an error in the test file regrading beta1 and beta2.

added 1 commit

• 03241e36 - Fix Adam optimizer unit test

Compare with previous version

added 1 commit

• a0dd82cc - Fix Adam optimizer unit test

Compare with previous version

added 1 commit

• c02cc1af - Fix Adam optimizer unit test

Compare with previous version

added 1 commit

• d298f23d - Fix Adam optimizer unit test

Compare with previous version

added 1 commit

• b76944f7 - Fix Adam optimizer unit test

Compare with previous version

approved this merge request

merged

mentioned in commit 51e3cf9e

Thank you very much for your efforts. It was a long road to success...