Skip to content

Initial study for designing agents, and function calling, as nodes in AI-Builder

What are AI Agents and Function Calling?

AI agents are intelligent systems autonomously sense, reason, and act to achieve user-defined goals. They can process user input, make decisions, and use external APIs and tools. All of the mentioned can occur iteratively.

Function calling allows LLMs to invoke external functions (APIs, tools, or scripts) instead of yielding plain text. This further helps access real-time data, perform computations, and interact with systems.

Agent interacts with tools to produce the necessary results. Function calling makes AI agents more robust and streamlined by combining LLM reasoning with real-world execution.

Multi-Agentic AI frameworks

  • LangGraph - LangGraph is a library for building stateful, multi-actor applications with LLMs, used to create agent and multi-agent workflows.
  • Phidata - Phidata is a framework for building multi-modal agents and workflows.
from phi.agent import Agent

agent = Agent(
    # Add functions or Toolkits
    tools=[...],
    # Show tool calls in the Agent response
    show_tool_calls=True
)
  • Microsoft AutoGen AutoGen offers a unified multi-agent conversation framework as a high-level abstraction of using foundation models. It features capable, customizable and conversable agents which integrate LLMs, tools, and humans via automated agent chat.

  • CrewAI Production-grade framework for orchestrating sophisticated AI agent systems.

What is Function Calling?

OpenAI Example

from openai import OpenAI
import json

client = OpenAI()

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current temperature for provided coordinates in celsius.",
        "parameters": {
            "type": "object",
            "properties": {
                "latitude": {"type": "number"},
                "longitude": {"type": "number"}
            },
            "required": ["latitude", "longitude"],
            "additionalProperties": False
        },
        "strict": True
    }
}]

messages = [{"role": "user", "content": "What's the weather like in Paris today?"}]

completion = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
)

To-do/Pointers

  • Define the intricacies in planning the common agentic workflow
  • Outline a goal-driven pipeline that ensures reasonable interaction in the workflow
  • Check for the available tools across all frameworks
  • Start defining new tools
  • Possibility for adding a separate section for tools in AI-Builder
  • How about a unified protobuf interface for agents?
syntax = "proto3";

// Request to call an external tool
message ToolRequest {
    string tool_name = 1;
    map<string, string> parameters = 2;  
}

// Response from a tool
message ToolResponse {
    string tool_name = 1;
    string status = 2;  
    string output = 3;  
}

// === Agent Service ===
// A query to an agent, which may involve LLM calls and tool use
message AgentQuery {
    string user_id = 1;
    string session_id = 2;
    string query = 3;
    repeated ToolRequest tool_calls = 4;  
}

// Response from an agent
message AgentResponse {
    string user_id = 1;
    string session_id = 2;
    string response_text = 3;
    repeated ToolResponse tool_results = 4;  
}

service AIAgent {
    rpc CallTool(ToolRequest) returns (ToolResponse);
    rpc AskAgent(AgentQuery) returns (AgentResponse);
}

AIAgent.drawio

References

Resource Description Link
Article What are Agents? Website
Article Importance of API, without Framework overhead Website
Article Agent Supervisor Website
Article Booking Multi-Agent System Website
Article Multi-Agent Swarm - Flight and Hotel Website
Article Farm trip agent Website
Article local-llm-function-calling Website
Article A team of agents: Orchestrator, Plan Critic, Researcher, and Parser Website
Article How LLM Agent works? Website
Article Building Effective Agents Website
Article Advanced OpenAI function call Website
Article Building Multi-Agent System Website
Article Multi agent flight booking crew Website
Article Multi agent flight booking crew Website
Article LLM tool-use Website
Article Production-Ready TripPlanner Multi-AI Agent Project Website

Research Papers

Resource Description Link
Papers Connected Papers Website
Papers Connected Papers Website
Paper AnyTool Website
Paper An LLM Compiler for Parallel Function Calling Website
Paper ToolFlow Website
Paper Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation Website

Tutorials

Resource Description Link
Code Azure OpenAI Service - function calling Website
Code llm-function-calling Website
Code Function Calling Mistral 7B Integration Website
Code OpenAI Advanced Function Website

Videos

Resource Description Link
Video What's next for AI agentic workflows ft. Andrew Ng of AI Fund Website
Edited by Swetha Lakshmana Murthy