Blogs

Tool Calling

Posted by Jesse JCharis

Feb. 22, 2025, 10:18 p.m.

Tool Calling

What Is Tool Calling?

Tool calling is a capability that allows large language models (LLMs) like GPT-4 or Claude to interact with external tools, APIs, or systems to perform tasks beyond text generation. Instead of relying solely on their internal knowledge, LLMs equipped with tool calling can execute code, retrieve real-time data, or trigger actions in the physical world—effectively transforming them into action-oriented AI agents.

For example:

• An LLM can call a weather API to answer “What’s the forecast in Tokyo tomorrow?”
• A coding assistant like GitHub Copilot might execute a script to test its generated code.

This bridges the gap between language understanding and practical functionality, enabling AI agents to act as dynamic problem solvers.

How Tool Calling Works

Tool calling typically follows these steps:
1️⃣ Intent Recognition: The LLM identifies when a user query requires external tools (e.g., “Book a flight to Paris” needs access to a travel API).
2️⃣ Tool Selection: The agent selects the appropriate tool (e.g., calendar app, database, payment gateway).
3️⃣ Execution: The tool processes the request (e.g., checking flight availability) and returns results.
4️⃣ Response Synthesis: The LLM interprets the results and delivers a human-readable answer.

Modern frameworks like LangChain, AutoGPT, and OpenAI’s Function Calling streamline this process by letting developers define tools and connect them seamlessly to LLMs.

Practical Examples of Tool Calling

1️⃣ Travel Planning Agent

• Task: Plan a trip based on user preferences (budget, dates).
• Tools Used: Flight/hotel booking APIs (e.g., Amadeus), Google Maps for directions.
• Workflow:
  • The LLM asks clarifying questions (“Do you prefer direct flights?”).
  • Calls APIs to compare prices and availability.
  • Returns an itinerary with booking links.

2️⃣ Customer Support Automation

• Task: Resolve a user’s billing issue.
• Tools Used: CRM systems (Salesforce), payment gateways (Stripe).
• Workflow:
  • The LLM authenticates the user via an API.
  • Retrieves their transaction history from a database.
  • Processes refunds or escalates complex cases to human agents.

3️⃣ Data Analysis Assistant

• Task: Analyze sales trends from a CSV file.
• Tools Used: Python libraries (Pandas), visualization tools (Matplotlib).
• Workflow:
  • The LLM generates Python code to clean and analyze data.
  • Executes the code in a sandboxed environment.
  • Returns insights as charts or summaries.

4️⃣ Smart Home Controller

• Task: Adjust home settings via voice command (“Turn off the lights”).
• Tools Used: IoT platforms (Google Home API).
• Workflow:
  • The LLM converts speech to structured commands ({“device”: “lights”, “action”: “off”}).
  • Sends instructions via APIs to smart devices.

5️⃣ E-commerce Agent

• Task: Help users find products (“Show red sneakers under $100”).
• Tools Used: Inventory databases (Shopify), recommendation engines.
• Workflow:
  • Queries product databases using natural language inputs translated into SQL.
  • Filters results based on price/color preferences using real-time APIs.

Frameworks Enabling Tool Calling

1️⃣ LangChain: Lets developers chain LLM calls with external tools (e.g., search engines or calculators) for tasks like research or math problem-solving.

2️⃣ OpenAI Function Calling: Allows GPT models to describe functions they need to execute (e.g., get_current_weather(location)), which developers can map to real-world APIs.

3️⃣ AutoGPT: An autonomous agent that recursively breaks down goals into subtasks, using web browsing and code execution tools for tasks like market research or content creation.

Challenges in Tool Calling

🔧 Security Risks: Granting AI access to tools like payment systems requires strict authentication and permission controls to prevent misuse (e.g., unauthorized purchases).

⏱️ Latency: Complex workflows involving multiple API calls can slow down response times—critical for real-time applications like chatbots.

🤖 Hallucination & Errors: LLMs may misuse tools if prompts are ambiguous (e.g., querying the wrong database column). Guardrails like input validation are essential.

🔄 Dependency Management: Tools like weather APIs may go offline; agents need fallback mechanisms for reliability.

The Future of Tool Calling

As LLMs grow more sophisticated, tool calling will enable AI agents to act as fully autonomous "doers":

• Personal AI assistants could manage your calendar and negotiate meeting times via email APIs.
• Healthcare agents might cross-reference patient symptoms with medical databases and book lab tests automatically.

Developers will increasingly focus on creating standardized interfaces (toolkits) for common tasks (e.g., sending emails or analyzing data), democratizing AI agent development.

Conclusion

Tool calling transforms LLMs from conversationalists into actionable problem solvers—whether booking flights or diagnosing software bugs By integrating external systems responsibly we unlock AI’s potential as a force multiplier for productivity What tools would you want your AI agent to master?

Below are Python code examples demonstrating tool calling with LLMs and AI agents using popular frameworks like OpenAI’s Function Calling and LangChain. These snippets illustrate how AI agents can interact with external tools/APIs.

Example 1: Basic Tool Calling with OpenAI Function Calling

Scenario: An AI agent uses an API to fetch real-time weather data based on user input.

import openai
import json

# Mock function to simulate weather API call
def get_current_weather(location: str) -> str:
    """Returns mock weather data for demonstration."""
    return json.dumps({
        "location": location,
        "temperature": "22°C",
        "forecast": "sunny"
    })

# Define available tools (APIs/functions)
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"}
                },
                "required": ["location"]
            }
        }
    }
]

# User query
user_query = "What's the weather in Tokyo?"

# Step 1: Ask GPT-4 if tool calling is needed
response = openai.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": user_query}],
    tools=tools,
    tool_choice="auto"
)

# Step 2: Extract tool call request from GPT's response
response_message = response.choices[0].message
tool_calls = response_message.tool_calls

if tool_calls:
    # Step 3: Execute the requested tool (weather API)
    function_name = tool_calls[0].function.name
    arguments = json.loads(tool_calls[0].function.arguments)
    
    if function_name == "get_current_weather":
        weather_data = get_current_weather(arguments["location"])
        
        # Step 4: Send results back to GPT for final response
        second_response = openai.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "user", "content": user_query},
                response_message,
                {"role": "tool", "content": weather_data}
            ]
        )
        print(second_response.choices[0].message.content)
else:
    print(response_message.content)

Output:
The current weather in Tokyo is sunny with a temperature of 22°C.

Example 2: LangChain Agent with Calculator Tool

Scenario: An AI agent uses a calculator tool to solve math problems.

from langchain import hub
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_community.tools import Tool
from langchain_openai import ChatOpenAI

# Define a calculator tool
def calculate(expression: str) -> str:
    """Evaluate arithmetic expressions."""
    return str(eval(expression))

calculator_tool = Tool(
    name="Calculator",
    func=calculate,
    description="Useful for solving math problems"
)

# Initialize agent with tools
llm = ChatOpenAI(model="gpt-3.5-turbo")
prompt = hub.pull("hwchase17/openai-tools-agent")
agent = create_openai_tools_agent(llm, [calculator_tool], prompt)
agent_executor = AgentExecutor(agent=agent, tools=[calculator_tool], verbose=True)

# Run query requiring tool use
response = agent_executor.invoke({
    "input": "What is 15 raised to the power of 2? Use a tool."
})
print(response["output"])

Output:

> Entering AgentExecutor chain...
Invoking: `Calculator` with `15 ** 2`

225

The result of 15 raised to the power of 2 is 225.

Example 3: Dynamic Code Execution Tool

Scenario: An AI agent writes and executes Python code dynamically (e.g., for data analysis).

import subprocess
import tempfile

def execute_python_code(code: str) -> str:
    """Execute Python code securely in a sandboxed environment."""
    try:
        # For safety, use temporary files and subprocess limits
        with tempfile.NamedTemporaryFile(mode='w', suffix='.py') as f:
            f.write(code)
            f.flush()
            result = subprocess.run(
                ["python", f.name],
                capture_output=True,
                text=True,
                timeout=10
            )
            return result.stdout.strip()
    except Exception as e:
        return f"Error: {str(e)}"

# Example usage (simulating an LLM-generated code snippet)
user_request = """
Generate a plot of y = sin(x) from 0 to 2π using matplotlib.
Save it as 'plot.png'.
"""

generated_code = """
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 2 * np.pi)
y = np.sin(x)

plt.plot(x, y)
plt.title('Sine Wave')
plt.savefig('plot.png')
"""

output = execute_python_code(generated_code)
print(f"Code executed successfully! Check 'plot.png'.")

Output:
A file plot.png containing the sine wave graph is generated.

Key Considerations

1️⃣ Security: Always sanitize inputs and sandbox code execution (e.g., Docker containers) when allowing dynamic tool calls.
2️⃣ Error Handling: Validate outputs from tools before feeding them back into LLMs (e.g., catching API timeouts).
3️⃣ Cost Optimization: Cache frequent tool calls (e.g., weather data) to reduce API costs.