Skip to content

Let SmartRAG use GPU if available

The SmartRAG node could detect if a GPU is available and accelerate creation of the embeddings see new ai-runner-gpu

Tasks

  • The nodes are able to detect GPU and assign resources to them
  • speed enhancement of rag-node in SmartRAG
    • The .to(device) searches for CUDA and move some processing layers to GPU.
    • Even when we do not use the .to(device) syntax, they run in CPU but still speed up --> this might be due to the powerful hosts, faster RAM etc. ( needs investigation )

image

Edited by Sangamithra Panneer Selvam