Initial study for designing agents, and function calling, as nodes in AI-Builder

What are AI Agents and Function Calling?

AI agents are intelligent systems autonomously sense, reason, and act to achieve user-defined goals. They can process user input, make decisions, and use external APIs and tools. All of the mentioned can occur iteratively.

Function calling allows LLMs to invoke external functions (APIs, tools, or scripts) instead of yielding plain text. This further helps access real-time data, perform computations, and interact with systems.

Agent interacts with tools to produce the necessary results. Function calling makes AI agents more robust and streamlined by combining LLM reasoning with real-world execution.

Multi-Agentic AI frameworks

LangGraph - LangGraph is a library for building stateful, multi-actor applications with LLMs, used to create agent and multi-agent workflows.

Phidata - Phidata is a framework for building multi-modal agents and workflows.

from phi.agent import Agent

agent = Agent(
    # Add functions or Toolkits
    tools=[...],
    # Show tool calls in the Agent response
    show_tool_calls=True
)

Microsoft AutoGen AutoGen offers a unified multi-agent conversation framework as a high-level abstraction of using foundation models. It features capable, customizable and conversable agents which integrate LLMs, tools, and humans via automated agent chat.
CrewAI Production-grade framework for orchestrating sophisticated AI agent systems.

What is Function Calling?

OpenAI Example

from openai import OpenAI
import json

client = OpenAI()

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current temperature for provided coordinates in celsius.",
        "parameters": {
            "type": "object",
            "properties": {
                "latitude": {"type": "number"},
                "longitude": {"type": "number"}
            },
            "required": ["latitude", "longitude"],
            "additionalProperties": False
        },
        "strict": True
    }
}]

messages = [{"role": "user", "content": "What's the weather like in Paris today?"}]

completion = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
)

To-do/Pointers

Define the intricacies in planning the common agentic workflow
Outline a goal-driven pipeline that ensures reasonable interaction in the workflow
Check for the available tools across all frameworks
Start defining new tools
Possibility for adding a separate section for tools in AI-Builder
How about a unified protobuf interface for agents?

syntax = "proto3";

// Request to call an external tool
message ToolRequest {
    string tool_name = 1;
    map<string, string> parameters = 2;  
}

// Response from a tool
message ToolResponse {
    string tool_name = 1;
    string status = 2;  
    string output = 3;  
}

// === Agent Service ===
// A query to an agent, which may involve LLM calls and tool use
message AgentQuery {
    string user_id = 1;
    string session_id = 2;
    string query = 3;
    repeated ToolRequest tool_calls = 4;  
}

// Response from an agent
message AgentResponse {
    string user_id = 1;
    string session_id = 2;
    string response_text = 3;
    repeated ToolResponse tool_results = 4;  
}

service AIAgent {
    rpc CallTool(ToolRequest) returns (ToolResponse);
    rpc AskAgent(AgentQuery) returns (AgentResponse);
}

References

Resource	Description	Link
Article	What are Agents?	Website
Article	Importance of API, without Framework overhead	Website
Article	Agent Supervisor	Website
Article	Booking Multi-Agent System	Website
Article	Multi-Agent Swarm - Flight and Hotel	Website
Article	Farm trip agent	Website
Article	local-llm-function-calling	Website
Article	A team of agents: Orchestrator, Plan Critic, Researcher, and Parser	Website
Article	How LLM Agent works?	Website
Article	Building Effective Agents	Website
Article	Advanced OpenAI function call	Website
Article	Building Multi-Agent System	Website
Article	Multi agent flight booking crew	Website
Article	Multi agent flight booking crew	Website
Article	LLM tool-use	Website
Article	Production-Ready TripPlanner Multi-AI Agent Project	Website

Research Papers

Resource	Description	Link
Papers	Connected Papers	Website
Papers	Connected Papers	Website
Paper	AnyTool	Website
Paper	An LLM Compiler for Parallel Function Calling	Website
Paper	ToolFlow	Website
Paper	Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation	Website

Tutorials

Resource	Description	Link
Code	Azure OpenAI Service - function calling	Website
Code	llm-function-calling	Website
Code	Function Calling Mistral 7B Integration	Website
Code	OpenAI Advanced Function	Website

Videos

Resource	Description	Link
Video	What's next for AI agentic workflows ft. Andrew Ng of AI Fund	Website

0 of 6 checklist items completed · Edited 1 month ago

Designs

Child items 0

No child items are currently assigned. Use child items to break down this issue into smaller parts.