Backward failure when using Data Provider with drop_last set to False
What commit version of aidge do you use
Current development version of core and CUDA modules
Problem description
When using DataProvider with drop_last set to false, the last batch is shorter than the previous ones. This requires that the tensor dimensions are correctly updated during training.
Unfortunately, when scheduler.backward() is called for this last batch, the following CUDA error is generated: "RuntimeError: CUDNN failure: CUDNN_STATUS_NOT_SUPPORTED (9)".
This error occurs in the Aidge::ReLUImpl_cuda::backward_ method, when cudnnActivationBackward is called using the (badly?) updated CUDA descriptors / dimensions for the input and output tensors.
Reproducible example code
Use the following example in the tutorial: /examples/tutorials/Learning/learn.ipynb
Where the following modifications have been done :
- Set BACKEND to "cuda",
- In the definition of "aidge_dataprovider", "drop_last" is set to "False",
- The break point after the 5 first iterations (at the end of the file) is removed.
Activity
Maybe related to !23 that was previously fixed
During the forward pass, input and *op.getOutput(0) have correct dimensions.
During the backward pass, *op.getInput(0), *op.getOutput(0) and output_grad have correct dimensions, but the dimension of *op.getInput(0)->grad() has not been updated.
Edited by Olivier AntoniIn the Aidge::Tensor class, it is unclear whether a tensor and its associated gradient tensor (if any) must always have the same dimensions. If so, this causes problems when calling Tensor::resize, because the tensor's dimensions are resized, but the gradient tensor's dimensions are unchanged. As a result, when the user calls scheduler.forward(), the gradient tensors are never resized (if already exist) to account for changes in the batch size.
I propose a fix that involves resizing gradient tensors in Aidge::Optimizer::resetGrad, but it might be better to modify Tensor::resize instead (to be discussed within the Aidge development team).
Edited by Olivier AntoniI think it would be better indeed to resize the gradient tensor in the
resize()function (if it has been initialized!). It seems to be a more generic solution and I don't see how resizing the tensor would not require to automatically resize the gradient...Edited by Olivier BICHLEROk, I propose this new merge request !397 that modifies resize() function for Tensor
mentioned in merge request !41 (closed)
added Bug 🐛 label
mentioned in merge request aidge_core!397 (merged)
Fixed in aidge_core!397 (merged).
