Large Language Models (LLMs) are powerful, but their knowledge is frozen at the time of their training. They can't check today's weather, look up a customer's order, or book a meeting in your calendar. LLM tool calling, also called function calling or tool use, solves this problem by allowing a model to request help from your application's code, connecting it to live data and real-world actions.

This guide explains how LLM tool calling works, moving beyond simple examples to cover the full execution loop, security considerations, and production best practices.

Key takeaways

  • The LLM only requests. The model's job is to generate a structured message asking to run a tool with specific arguments. Your application is always responsible for parsing, validating, and executing the actual code (Source: OpenAI API).
  • Security is your responsibility. Treat every argument from the model as untrusted user input. Your application code must validate inputs and enforce permissions before running any function, especially one that changes data (Source: vLLM documentation).
  • Clear descriptions are crucial. Clear, specific descriptions of a tool's purpose are crucial for the model to reliably select the correct tool and generate appropriate arguments.
  • It's a multi-step conversation. Tool calling isn't a single request and response. It's a loop: you define tools, the model requests a call, your app executes it, your app sends the result back, and the model uses that result to generate the final answer or make additional calls (Source: Google Cloud).
  • Support varies by provider. Tool-calling features differ across models and APIs. A portable application should always verify the target provider's conventions for schema support, tool-choice controls, and response formats (Source: vLLM documentation).
  • Tool calling enables agents, but is not an agent. This mechanism is a fundamental building block for creating AI agents. However, a true agent requires more complex logic for planning, memory, and state management, which are built around the core tool-calling interface (Source: OpenAI Tools Guide).

How Does LLM Tool Calling Work? The Step-by-Step Execution Loop

LLM tool calling works through a structured, multi-turn conversation between your application and the model. Your code remains in full control of execution; the model generates a structured request for an action to be taken. This clear separation is crucial for building safe and reliable applications. The model never runs any of your code directly (Source: OpenAI API).

Here is the complete process, step by step:

  1. Define Your Tools. You describe your available functions to the model. Each tool definition includes a name, a clear description of what it does, and a schema for its parameters. This schema, often using a format compatible with JSON Schema, specifies the arguments the function accepts, their data types (e.g., string, number), and which are required (Source: Google Cloud).

  2. Send the Request. Your application sends the user's prompt and the conversation history to the model. Critically, you also include the list of tool definitions from Step 1 in this API call.

  3. Model Generates a Tool Call. The model analyzes the user's request in the context of the tools you provided. If it determines that one of your tools can help answer the query, it doesn't respond with natural language. Instead, it produces a structured request that identifies the tool to run and the arguments it needs. The exact format, such as a tool_calls object, varies by provider (Source: NVIDIA NIM).

  4. Parse and Validate in Your Application. Your application receives this structured request. Your code must then parse the arguments (which are usually in a JSON string). This is a critical security checkpoint. You must validate the arguments against your business rules and check if the user is authorized to perform this action. Treat these arguments like any other input from an external user.

  5. Execute the Tool. Once validated, your application calls the corresponding function in your codebase. This might involve querying a database, calling another API, or performing a calculation. The LLM is not involved in this step at all; the execution happens entirely within your application's environment.

  6. Return the Result. After your function runs, you send its output back to the model in a new API call. This message must include an identifier, such as a tool_call_id, to link the result to the original request.

  7. Model Generates the Final Response. The model receives the tool's output. It then synthesizes this new information and generates a final, natural-language response for the user. Alternatively, it might decide it needs more information and emit further tool calls, repeating the loop.

A Complete Tool-Calling Example: From User Query to Final Answer

To make the process concrete, let's trace a simple user request from start to finish. Imagine we have a function called getCurrentWeather (Source: Typo3 Docs).

Step 1: Define the Tool

First, we define the tool using a JSON Schema-like structure. This tells the model the tool's name, purpose, and parameters.

JSON
{
  "type": "function",
  "function": {
    "name": "getCurrentWeather",
    "description": "Get the current weather for a specific location",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {
          "type": "string",
          "description": "The city and state, e.g., San Francisco, CA"
        },
        "unit": {
          "type": "string",
          "enum": ["celsius", "fahrenheit"],
          "description": "The unit of temperature"
        }
      },
      "required": ["location"]
    }
  }
}

Step 2: User Asks a Question

The user sends a prompt to our application.

User: "What's the weather like in Boston?"

Our application sends this prompt to the model, along with the tool definition from Step 1.

Step 3: Model Requests a Tool Call

The model recognizes that the user's query matches the purpose of the getCurrentWeather tool. Instead of answering directly, it returns a structured request. The example below shows a common format using a tool_calls object.

JSON
{
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "getCurrentWeather",
        "arguments": "{\"location\": \"Boston, MA\"}"
      }
    }
  ]
}

Note: The model inferred "Boston, MA" and left the optional unit parameter blank.

Step 4 & 5: Application Executes the Tool

Our application code receives this object.

  1. It checks that getCurrentWeather is a valid tool.
  2. It parses the arguments string into a JSON object.
  3. It validates the location argument.
  4. It calls the internal getCurrentWeather("Boston, MA") function.
  5. Let's say this function returns: {"temperature": "52", "unit": "fahrenheit", "condition": "Cloudy"}.

Step 6: Application Returns the Result

Our application sends the result back to the model, making sure to include the original identifier (tool_call_id in this case) to link it to the request.

Tool Message: {"tool_call_id": "call_abc123", "role": "tool", "name": "getCurrentWeather", "content": "{\"temperature\": \"52\", \"unit\": \"fahrenheit\", \"condition\": \"Cloudy\"}"}

Step 7: Model Generates Final Response

The model receives the weather data. Now it has the information it needs to answer the user's original question in a helpful, conversational way.

Final Assistant Response: "The current weather in Boston is 52°F and cloudy."

How Does the Model Decide Which Tool to Call?

The model's choice of a tool is a model prediction, not a formally verified plan. It analyzes the user's prompt and compares it to the tool definitions you provide, choosing the tool that seems most likely to help fulfill the request based on its training.

The key factors influencing its decision are:

  • Tool Description: A clear, concise, and accurate description of what the tool does is crucial. For example, "Searches for customer order details by order ID" is much better than "Gets data." The model matches the intent of the user's prompt to the text in your descriptions.
  • Tool Name: A descriptive name like find_order_by_id provides another strong signal to the model.
  • Parameter Schema: The names and descriptions of the arguments also guide the model. If a user says, "Find the order for customer 12345," and you have a tool with a customer_id parameter, the model is more likely to select it.

You can also override the model's default behavior. Many APIs provide a tool_choice parameter that lets you force a specific mode (Source: NVIDIA NIM):

  • auto: The default. The model decides whether to call a tool or not.
  • none: Prevents the model from calling any tool, forcing it to generate a text-based response.
  • required: Forces the model to call at least one of the provided tools.
  • Specific tool: You can force the model to use a particular tool, for example { "type": "function", "function": { "name": "getCurrentWeather" } }.

Tool Calling vs. RAG vs. Standard API Calls

Tool calling is a general-purpose technique. It helps to understand how it relates to other common patterns like Retrieval-Augmented Generation (RAG) and traditional API calls.

Feature LLM Tool Calling Retrieval-Augmented Generation (RAG) Standard API Call
Decision Maker The LLM decides which tool to call based on the user's prompt and provided tool descriptions. Often implemented via an LLM tool call, where the LLM decides to trigger a "retrieve" tool. The application developer writes explicit code to call a specific API endpoint.
Input/Output Structured request from LLM (e.g., JSON) -> Application executes -> Structured result back to LLM. Unstructured query -> Retrieval system finds relevant documents -> Documents + query sent to LLM. Application sends structured request -> API returns structured data.
Flexibility High. The LLM can choose from many different tools or none at all, based on the conversation. Medium. It's a specific pattern focused on augmenting a prompt with external documents. Low. The code path is fixed; it always calls the same API endpoint.
Best For... Integrating LLMs with any external system: databases, APIs, or internal business logic. Answering questions using a specific body of knowledge, like internal company documents or technical manuals. Deterministic, predictable integrations between software services.

In short, RAG is a very common use case of tool calling. You implement RAG by creating a retrieve_documents tool and letting the LLM call it when the user asks a question that can be answered from your knowledge base.

What Are the Common Mistakes and Limits of LLM Tool Calling?

While powerful, LLM tool calling introduces new complexities. Avoiding common pitfalls is essential for building a reliable and secure application.

Design Pitfalls

  • Vague Descriptions: Writing unclear or overlapping tool descriptions (e.g., two tools that could both handle the same user request) confuses the model and leads to unreliable behavior.
  • Overly Broad Tools: A single tool that does too many things (e.g., a manage_user tool that can create, delete, and update) is harder for the model to use correctly and poses a greater security risk. Prefer smaller, single-purpose tools.
  • Too Many Tools: Providing dozens of tools in a single request can overwhelm the model, increase latency, and consume a large number of tokens. Group tools contextually and only provide the ones relevant to the current task.

Security Pitfalls

  • Trusting Model Arguments: Never trust that the arguments provided by the model are safe. Always validate them. For example, check that a user ID is in the correct format, that a number is within an expected range, and that a string doesn't contain malicious code. This is related to the broader risk of Prompt Injection: The Top LLM Security Risk Explained.
  • Executing Destructive Actions Without Confirmation: If a tool can delete data, send money, or post public content, always implement a human confirmation step. Your application should ask the user "Are you sure you want to delete order #54321?" before executing the action.
  • Ignoring Authorization: The model doesn't know who your user is or what they are allowed to do. Your application code that executes the tool must check the user's permissions before running. Just because the model requests to delete a user doesn't mean the current user has permission to do so.

Execution Pitfalls

  • Not Handling Errors: A tool might fail. An external API could be down, a database query could time out, or the arguments could be invalid despite initial checks. Your code must catch these errors and return a clear error message to the model so it can explain the problem to the user.
  • Creating Infinite Loops: Be careful with autonomous loops. A model might repeatedly call the same failing tool. Implement circuit breakers, retry limits, and require user intervention if a task is not progressing.
  • Assuming Full JSON Schema Support: Providers and model servers may support only subsets of JSON Schema. Always check the documentation for your specific model to see which keywords and structures are supported.
  • Returning Too Much Data: A tool that returns thousands of database rows will be expensive and slow. Summarize or paginate data before sending it back to the model's context window.

What to learn next

Sources and further reading