What is Function Calling?

LLM function calling is the ability of large language models (LLMs) to invoke external functions or APIs in response to user queries. For example, if a user asks for the current weather in a specific location, the LLM can use function calling to call a weather API, fetch real-time data, and present it in a structured response.

This capability allows LLMs to enhance their functionality by integrating with other systems, services, or databases to provide real-time data, perform specific tasks, or trigger actions.

How Function Calling Works

• User Query: A user asks the LLM something that requires external data or specific action (e.g., “Check the weather in New York” or “Book an appointment”).
• Function Identification: The LLM identifies that the query requires a specific function to be called. For instance, if the user asks for the weather, the model recognizes that it needs to call a weather API rather than generate a general response.
• Parameter Extraction: The LLM analyzes the user’s query to extract the required parameters (e.g., location, date, or other variables). For example, in the weather query, “New York” would be extracted as the location parameter.
• Call the Function: The LLM triggers an external function or API with the appropriate parameters. This could involve fetching live data, performing a task (e.g., making a booking), or retrieving information that is outside the LLM’s static knowledge.
• Return the Result: The function returns the result (such as the current weather data), which the LLM incorporates into its response back to the user.

A process flow of the Voice Agent API with function calling.

Settings Configuration

Here is an incomplete Settings Configuration object showing what properties can be added to configure function calling with Deepgram’s Voice Agent API. (See a more complete Settings Configuration object in the relevant documentation.)

1
{
2
  "type": "SettingsConfiguration",
3
  ... // other settings configuration fields
4
  "agent": {
5
    "think": {
6
      "functions": [
7
        {
8
          "name": "", // function name
9
          "description": "", // tells the agent what the function does, and how and when to use it
10
          "url": "", // the endpoint where your function will be called
11
          "headers": [
12
            {
13
              "key": "authorization",
14
              "value": ""
15
            }
16
          ],
17
          "method": "post", // the http method to use when calling the function
18
          "parameters": {
19
            "type": "object",
20
            "properties": {
21
              "item": { // the name of the input property
22
                "type": "string", // the type of the input
23
                "description":"" // the description of the input so the agent understands what it is
24
              }
25
            },
26
            "required": ["item"] // the list of required input properties for this function to be called
27
          }
28
        }
29
      ]
30
    }
31
  }
32
}

Not all of the fields are required in every function calling example. If the function will run client-side and you do not need to make a request to your server, you may not need fields such as url, headers, or method, for example. To help visualize this, here is the configuration settings for the client-side weather example:

1
functions: [
2
        {
3
          name: 'get_weather',
4
          description: 'Get the current weather for a specific location',
5
          parameters: {
6
            type: 'object',
7
            properties: {
8
              location: {
9
                type: 'string',
10
                description: 'The city or location to get weather for',
11
              },
12
            },
13
            required: ['location'],
14
          },
15
        },
16
      ],

The function itself would be called when the user interacting with the LLM requests weather information for a specific location:

1
export const getWeather = async (location: string): Promise<string | null> => {
2
  const apiKey = import.meta.env.VITE_OPENWEATHER_API_KEY;
3
4
  try {
5
    const response = await fetch(
6
      `https://api.openweathermap.org/data/2.5/weather?q=${location}&appid=${apiKey}`
7
    );
8
9
    if (!response.ok) {
10
      throw new Error('Failed to fetch weather data');
11
    }
12
13
    const data = await response.json();
14
15
    return `The current weather in ${data.name} is ${data.weather[0].description} with a temperature of ${data.main.temp}°K.`;
16
  } catch (err) {
17
    console.error(err);
18
    return null;
19
  }
20
};

Server and Client Function Calling Messages

Function calling messages are exchanged between the client and Deepgram’s Voice Agent API server through the websocket.

Server Messages (Sent by Deepgram)

• FunctionCalling – Indicates that the agent is invoking a function. Read more.
• FunctionCallRequest – Requests execution of a function with extracted parameters. Read more.
• FunctionCalling (Tool Response) – Returns the function’s output after execution. Read more.

Example:

1
{
2
  "type": "FunctionCallRequest",
3
  "function_name": "do_math",
4
  "function_call_id": "7433439b-c4b6-4369-8ce8-4124f6a98a1d",
5
  "input": { "numbers": ["2", "2"], "operation": "add" }
6
}

Client Messages (Sent by your application)

• FunctionCallResponse – Sent after executing the function to return the result.

Example:

1
{
2
  "type": "FunctionCallResponse",
3
  "function_call_id": "7433439b-c4b6-4369-8ce8-4124f6a98a1d",
4
  "output": "4"
5
}

Message Flow

1. User Query → The agent determines a function is needed.
2. Server Sends FunctionCallRequest → Requests function execution.
3. Client Executes & Sends FunctionCallResponse → Returns the function result.
4. Server Uses Response → The agent continues the conversation.

What’s Next

• Function Call Response