Configure TGI Integration: FEC VM Connectivity to LLM-Hosting API

Text Generation Inference (TGI) [1] is a toolkit designed for deploying and serving Large Language Models (LLMs) in production environments. With TGI, we can operate an LLM as a service, enabling its utilization across myriad clients.

Tasks :

• Prepare the selected LLM model(s) in a format compatible with TGI's requirements for deployment.
• Coordinate with the LLM-hosting team to add the wrapped LLM model (TGI compatible) to their list of available models.
• Create a simple pipeline to export the model in AI-Builder and test it.

References

1. https://github.com/huggingface/text-generation-inference

added to epic &20

changed the description

changed the description

changed the description

marked the checklist item Prepare the selected LLM model(s) in a format compatible with TGI's requirements for deployment. as completed

marked the checklist item Coordinate with the LLM-hosting team to add the wrapped LLM model (TGI compatible) to their list of available models. as completed

• The new LLM-hosting platform (GenAI Gateway) is only accessible within the FHG intranet and, hence, cannot be used in Graphene.
• Unique identifier issue in Fountain API (demo2) platform: Since the meta-llama/Meta-Llama-3-8B-Instruct is already hosted there, we need a unique identifier. This might not be possible with TGI.
• We can still host the meta-llama/Meta-Llama-3-8B. But meta-llama/Meta-Llama-3-8B-Instruct could be a better option for Q&A pipelines.

added Work In Progress label

Throughout this ticket, we explored frameworks such as TGI and vLLM, successfully hosting multiple LLMs on the GenAI platform as one of the nodes. However, no LLMs are hosted in this FEC, as we are now fully utilizing LLMs from the GenAI platform.

closed

removed Work In Progress label