Implement Dynamic Docker Container Generation for Various LLMs in the AI-Builder

The core idea is to dynamically create and manage individual Docker containers based on model names available from the LLM hosting platform and other sources. Users can drag and drop them into their workflows upon successful creation and onboarding in the AI builder.

Initial Design in AI-Builder for the LLMs:

Steps for the above defined task

1. Build Dynamic Dockerfiles and Registry Management
   
2. Onboarding the Docker containers to the AI-Builder

Drag-and-Drop for LLMs

Discussion Points:

1. Automating the dynamic Dockerfile generation and registry management process.
2. Evaluating the possibility of automating the onboarding process for individual models.
3. Creating a dedicated section within the AI-Builder’s Design Studio for easy access and management of these models.

added to epic &20

changed title from Implement Dynamic Docker Container Generation for Multiple Models in the LLM Hosting Platform to Implement Dynamic Docker Container Generation for Various LLMs in the AI-Builder

changed the description

Currently, the LLMs are on-boarded to the AI-Builder using the old script to utilize the hosted LLMs.

The protobuf file definition also currently follows the unified protobuf definition as defined here, ticket: Protobuf Definition - Unified Interface Ticket

To-Do List:

• Test the LLMs with the RAG-Construction Toolkit.
• Adapt to the new LLM-Hosting SDK once it is publicly available.
• Add OpenAI Docker containers with the new Graphene Project Key.
• Refactor the code for the Docker container generation script.
• Make streaming available by adapting the protobuf definition accordingly or using it from the SDK.

Current Issues:

• Long Docker tags are causing issues

changed the description

marked this issue as related to #35 (closed)

added Work In Progress label

The current state of the LLM containers:

A simple user interface is created as shown below,

Possible Improvizations

1. Handle Streaming Capabilities:
   Allow continuous and interactive conversations without requiring users to input everything simultaneously.
2. Session Management:
   Enable user sessions for personalized records and clearing the conversation when required.
3. History Management:
   Maintain a context-aware history of interactions to ensure continuity and coherence in ongoing conversations.
4. Handling the Number of Requests and Responses:
   Handling increased number of requests and responses - setting a limit on this if required

Update as on 27_08_2024

• Integration with the new SDK is in progress, which includes adapting and refactoring the docker container generation code.
• API keys have been configured with the playground as env variables for secure access.
• CodeLlama is on-boarded.

Update as of 03_09_2024

• SDK integrated and models are published on the AI-Builder.
• The LLM containers now support basic session and history management, which is still a work in progress.

The current status of the LLM containers is as mentioned in the following repository - https://gitlab.eclipse.org/eclipse/graphene/tutorials/-/tree/main/llm-containers-hub?ref_type=heads

This ticket has pending work, including streaming capabilities, session, and history management, request handling optimization, chatbot improvements, model card integration, and code formatting for uniformity. Please refer to the following ticket #53 for the remaining work as mentioned.

closed

removed Work In Progress label

mentioned in issue #35 (closed)