Configure the Voice Agent

To configure your Voice Agent, you’ll need to send a Settings message immediately after connection. This message configures the agent’s behavior, input/output audio formats, and various provider settings.

Settings Overview

The Settings message is a JSON object that contains the following fields:

Settings

Audio

Agent Settings

agent.context

• The agent.context object allows you to provide conversation history to the agent when starting a new session. This is useful for continuing conversations or providing background context.
• The agent.context.messages array contains conversation history entries, which can be either conversational messages or function calls.
• Conversational messages have the format: {"type": "History", "role": "user" | "assistant", "content": "message text"}
• Function call messages have the format: {"type": "History", "function_calls": [{"id": "unique_id", "name": "function_name", "client_side": true/false, "arguments": "json_string", "response": "response_text"}]}
• Use this feature to maintain conversation continuity across sessions or to provide the agent with relevant background information.
• To disable function call history, set settings.flags.history to false in the Settings message.

Agent - Listen Settings (STT)

agent.listen.provider.model

• To use Flux use flux-general-en, or flux-general-multi for multilingual support.
• When using flux-general-multi, set agent.listen.provider.language_hints to an array of BCP-47 language codes to bias toward expected languages. See Flux Multilingual & Language Prompting.
• Refer to the language availability for Flux to understand the language options.

agent.listen.provider.language

• Choose your language parameters based on your use case:

  • If you know your input language, specify it directly in agent.listen.provider.language for the best recognition accuracy.
  • If you expect multiple languages or are unsure, use multi in agent.listen.provider.language for flexible language support (Nova models), or use flux-general-multi with language_hints for Flux-based multilingual support.

• Refer to our supported languages to ensure you’re using the correct model (Flux, Nova-3, or Nova-2) for your selected language.

• For detailed multilingual setup, see Multilingual Voice Agents.

agent.listen.provider.eot_threshold, agent.listen.provider.eager_eot_threshold, and agent.listen.provider.eot_timeout_ms

These parameters control Flux end-of-turn detection and are only available when using Flux models with the v2 API (agent.listen.provider.version set to v2).

• eot_threshold sets the confidence required to trigger an EndOfTurn event. Higher values reduce false positives but increase latency. Defaults to 0.7.
• eager_eot_threshold enables eager end-of-turn detection, triggering EagerEndOfTurn events before the user fully finishes speaking. This reduces end-to-end latency but increases LLM calls. Must be less than or equal to eot_threshold.
• eot_timeout_ms sets a hard timeout in milliseconds — a turn finishes when this much time has passed after speech, regardless of EOT confidence. Defaults to 5000.
• All three parameters can be updated during a conversation using the UpdateListen message.
• For detailed tuning guidance, see the Flux end-of-turn configuration.

agent.listen.provider.smart_format

• The agent.listen.provider.smart_format setting is only available for Deepgram providers.
• When set to true, Deepgram applies smart formatting to improve transcript readability.
• Useful for UI-based apps that display Agent transcripts on screen, as it formats the text for better readability.
• When set to false, Deepgram does not apply smart formatting.
• The default value is false.
• When using Flux, you cannot use smart_format.

Agent - Think Settings (LLM)

agent.think.provider.reasoning_mode

• The reasoning_mode parameter maps to OpenAI’s reasoning_effort parameter.
• Accepts low, medium, or high. Higher values allow the model to spend more tokens reasoning before responding, which can improve accuracy on complex tasks.
• Only supported with OpenAI reasoning models (e.g., gpt-5, gpt-5-mini).

agent.think.context_length

• Using max will set the context length to the maximum allowed based on the LLM provider you use. If the total context exceeds the model’s maximum, truncation is handled by the LLM provider.
• Increasing the context length may help preserve multi-turn conversation history, especially when verbose function calls inflate the total context.
• All characters sent to the LLM count toward the context limit, including fully serialized JSON messages, function call arguments, and responses. System messages are excluded and managed separately via agent.think.prompt.
• The default context length set by Deepgram is optimized for cost and latency. It is not recommended to change this setting unless there’s a clear need.

Agent - Speak Settings (TTS)

agent.speak.provider.version

• Applies to Deepgram TTS providers only. Selects the Deepgram text-to-speech model family.
• Set v2 to use Flux TTS with a flux-{voice}-{language} model such as flux-alexis-en.
• Set v1 (the default when omitted) to use Aura models such as aura-2-thalia-en.
• Flux TTS voices are served only on v2 and Aura voices only on v1, so change version and model together.

agent.speak.provider.language

• Currently, multi is only supported in agent.speak.provider.language with Eleven Labs TTS, OpenAI TTS, or Cartesia TTS.
• Refer to our supported languages to ensure you’re using the correct model (Flux, Nova-3, or Nova-2) for your selected language.
• For detailed multilingual setup, see Multilingual Voice Agents.

TTS Controls - Speed, Pronounciation, Pause / Pacing

• Use agent.speak.provider.speed to control speed for each session
• Leverage the prompt (agent.think.prompt) for pronunciation, pause, and pacing controls. For detailed recommendations, see Voice Agent TTS Controls.

Full Example

Below is an in-depth example showing all the available fields for Settings with all the optional fields for individual provider specific settings.

1
{
2
  "type": "Settings",
3
  "tags": ["order", "customer_service"],
4
  "experimental": false,
5
  "mip_opt_out": false,
6
  "flags": {
7
    "history": true
8
  },
9
  "audio": {
10
    "input": {
11
      "encoding": "linear16",
12
      "sample_rate": 24000
13
    },
14
    "output": {
15
      "encoding": "mp3",
16
      "sample_rate": 24000,
17
      "bitrate": 48000,
18
      "container": "none"
19
    }
20
  },
21
  "agent": {
22
    "context": {
23
      "messages": [
24
        {
25
          "type": "History",
26
          "role": "user",
27
          "content": "What's my order status?"
28
        },
29
        {
30
          "type": "History",
31
          "function_calls": [
32
            {
33
              "id": "fc_12345678-90ab-cdef-1234-567890abcdef",
34
              "name": "check_order_status",
35
              "client_side": true,
36
              "arguments": "{\"order_id\": \"ORD-123456\"}",
37
              "response": "Order #123456 status: Shipped - Expected delivery date: 2024-03-15"
38
            }
39
          ]
40
        },
41
        {
42
          "type": "History",
43
          "role": "assistant",
44
          "content": "Your order #123456 has been shipped and is expected to arrive on March 15th, 2024."
45
        }
46
      ]
47
    },
48
    "listen": {
49
      "provider": {
50
        "type": "deepgram",
51
        "model": "flux-general-en",
52
        "version": "v2",
53
        "keyterms": ["hello", "goodbye"],
54
        "eot_threshold": 0.8,
55
        "eager_eot_threshold": 0.5,
56
        "eot_timeout_ms": 5000
57
      }
58
    },
59
    "think": {
60
      "provider": {
61
        "type": "open_ai",
62
        "model": "gpt-4o-mini",
63
        "temperature": 0.7,
64
        "reasoning_mode": "medium"
65
      },
66
      "endpoint": {
67
        "url": "https://api.example.com/llm",
68
        "headers": {
69
          "authorization": "Bearer {{token}}"
70
        }
71
      },
72
      "prompt": "You are a helpful AI assistant focused on customer service.",
73
      "context_length": 15000,
74
      "functions": [
75
        {
76
          "name": "check_order_status",
77
          "description": "Check the status of a customer order",
78
          "parameters": {
79
            "type": "object",
80
            "properties": {
81
              "order_id": {
82
                "type": "string",
83
                "description": "The order ID to check"
84
              }
85
            },
86
            "required": ["order_id"]
87
          },
88
          "endpoint": {
89
            "url": "https://api.example.com/orders/status",
90
            "method": "post",
91
            "headers": {
92
              "authorization": "Bearer {{token}}"
93
            }
94
          }
95
        }
96
      ]
97
    },
98
    "speak": {
99
      "provider": {
100
        "type": "deepgram",
101
        "version": "v1",
102
        "model": "aura-2-thalia-en",
103
        "speed": 1.0,
104
        "volume": 1.0,
105
        "model_id": "1234567890",
106
        "voice": {
107
          "mode": "id",
108
          "id": "a167e0f3-df7e-4d52-a9c3-f949145efdab"
109
        },
110
        "language": "en",
111
        "engine": "standard",
112
        "credentials": {
113
          "type": "IAM",
114
          "region": "us-east-1",
115
          "access_key_id": "{{access_key_id}}",
116
          "secret_access_key": "{{secret_access_key}}",
117
          "session_token": "{{session_token}}"
118
        }
119
      },
120
      "endpoint": {
121
        "url": "https://api.example.com/tts",
122
        "headers": {
123
          "authorization": "Bearer {{token}}"
124
        }
125
      }
126
    },
127
    "greeting": "Hello! How can I help you today?"
128
  }
129
}

Next Steps

• Voice Agent Message Flow for the correct message flow when building a Voice Agent client.