Enhance LLM Container's Capabilities - Integration of Unified LLM Interface
This ticket focuses on unifying the existing LLM interfaces. Currently, there are two separate protobuf interfaces: one used for applications like SmartRAG and Document Generation, and another for tool definitions in Event Planner. To reduce redundancy and streamline development, the goal is to merge these into a single unified LLM interface. Following this, we will generate dedicated Docker containers.
Tasks:
- Define a single, unified LLM protobuf file.
-
Refactor service code by:
- Combining the
InstructLLMStreamandGetUserQueryservices. - Enabling the Chat Interface only for direct LLM calls; disable it otherwise to avoid confusion in UI selection for applications such as SmartRAG or others.
- Implement a mechanism to detect and handle repeated identical queries.
- Combining the
- Improve the Chat Interface for better usability.
- Develop a single-node pipeline as an alternative environment for testing LLM integrations.
- Provide a way to set LLM parameters such as temperature and others dynamically.
/LLM
│
├── proto/
│ ├── unified.proto
│
├── apps/
│ ├── direct_llm_application.py
│ ├── rag/
│ │ └── rag_application.py
│ ├── readme/
│ │ └── readme_application.py
│ └── lettergen/
│ └── letter_gen_application.py
│
├── helpers/
│ └── llm_utils.py # Helper classes like LLMHelper used in tool-based server
│
├── services/
│ ├── llm_service.py # General LLMServiceServicer (from your first code)
│ ├── tool_agent_service.py # Tool-based LLMServicer adapted for unified.proto
│ └── unified_server.py # Script to run both services in one gRPC server
│
├── application_manager.py # Factory for managing app instances (if applicable)
├── app.py # For app_run or other background tasks
├── requirements.txt # Dependencies
└── README.md # Project documentation