Support Int types in backend_cuda

Currently, backend_cuda only supports 3 types: half(float16), float(float32) and double(float64).
Some users are asking for Int types support which is not easy for all operators:

Operators with Cublas kernels:

For the moment we only use cublasGemm and from the documentation, it supports int8 and int32 so these operators won't create a problem.

Operators with Custom kernels:

These operators have custom kernels in a .cu file. For these operators, supporting Int types is rather easy.

Operators with Cudnn kernels:

These operators' kernels are based on Cudnn functions. For these operators the task won't be easy since for some cudnn functions, int types are not supported.
example:

• cudnnPoolingForward (does not support int tensors)
• cudnnConvolutionForward (only supports CUDNN_DATA_INT8 but not CUDNN_DATA_INT32)
• ...

So what should we do for Cudnn-based operators? Will there be a specific custom kernel for int types besides the cudnn one for float types?

added StatusTODO label

removed StatusTODO label

added Discussion 💬 label

Linked to #35

not really, I am editing this issue, wait

Well it kinda is since you need to add registration :)

changed the description

assigned to @hrouis and @olivierbichler

unassigned @olivierbichler

AFAIK, it is mostly support for int8 that is asked so it won't pose a problem for now with CUDNN?

But yeah we would need to write custom kernel for these kernels in int32.

I didn't dive deep into all the cudnn func we use but Pooling does not support any int type.
This would make the backend cuda even harder to maintain

Also, another problem after adding dilation supp for pooling on cpu, I found out that cudnn does not support dilation ...
So I believe it is worth it for some operator (pooling for instance) to replace cudnn with custom kernel even if they are a bit slower

We can use ImplSpec to select cudnn impl if no dilation (most of the time?) and a custom impl if dilation.