Build a Real-Time Phone Voice Agent with Twilio and the Deepgram Voice Agent API

This guide walks through a complete, real-time phone agent: a caller dials a Twilio number, speaks naturally, and holds a conversation with an AI that can be interrupted mid-sentence, the way people talk to each other.

The architecture is small. Twilio streams the call audio to your server, and your server bridges that audio to the Deepgram Voice Agent API, a single WebSocket that runs the entire conversation: speech-to-text, the language model, text-to-speech, and turn-taking. By the end you have a working agent you can call from any phone, plus a clear view of how little code stands between the caller and a fully managed voice pipeline. That includes barge-in, the ability to talk over the agent and have it stop instantly.

How the agent works

A voice agent runs a tight loop: audio in, understanding, a reply, speech out, fast enough to feel like a conversation. The Voice Agent API owns that loop. Your server only moves audio between two sockets and reacts to one control signal, as the diagram shows.

The implementation is a single bridge running per call. It forwards the caller’s audio into the agent with send_media, forwards the agent’s audio back to Twilio as media frames, and listens for one event, UserStartedSpeaking, that tells it the caller has interrupted. Everything that would otherwise be a separate moving part lives inside Deepgram.

Which Twilio + Deepgram integration do you need?

This guide builds the full conversational agent on the unified Voice Agent API. If you only need one half of the pipeline, two focused guides cover those cases:

You want to…	Use	TwiML
Hold a two-way conversation (STT + LLM + TTS in one socket)	This guide — Voice Agent API	`<Connect><Stream>`
Transcribe a call in real time, nothing spoken back	Twilio and Deepgram STT	`<Start><Stream>`
Play generated speech into a call, no listening	Twilio and Deepgram TTS	`<Connect><Stream>`

Most phone-agent projects want this guide — the Voice Agent API handles listening, thinking, and speaking together, so you don’t wire STT and TTS by hand.

Before you begin

You need the following accounts, keys, and tools.

A Twilio account with a voice-capable phone number.
A Deepgram API key. Sign up free and you start with credits, no card required. This single key covers the whole agent.
Python 3.10 or later.
A tunneling tool to expose your local server to Twilio. This guide uses ngrok.

You don’t need a separate LLM provider key. The Voice Agent can use OpenAI, Anthropic, or another provider as its language model, but Deepgram manages that connection and bills it through your Deepgram account, whichever provider you select.

Install ngrok and authenticate it once with the token from your ngrok dashboard.

$ # macOS
$ brew install ngrok
$ ngrok config add-authtoken <YOUR_NGROK_AUTHTOKEN>

Step 1: Set up the project

Clone the companion repository, deepgram-devs/twilio-voice-agent, which holds the complete app.py, requirements.txt, and .env.example referenced throughout this guide.

$ git clone https://github.com/deepgram-devs/twilio-voice-agent.git
$ cd twilio-voice-agent

With the code in place, install the dependencies and set up configuration. The dependency list is short, because Deepgram does the heavy lifting.

$ pip install -r requirements.txt        # fastapi, uvicorn, deepgram-sdk, python-dotenv
$ cp .env.example .env                   # then fill in your Deepgram key + public host

The .env file holds just two values your application reads on startup.

$ DEEPGRAM_API_KEY=...
$ PUBLIC_HOSTNAME=your-host.ngrok-free.app   # the public host Twilio reaches, no scheme

Speech-to-text, the language model, and text-to-speech all run inside the Voice Agent, which your application reaches through the official Deepgram Python SDK (deepgram-sdk).

Verify: running python -c "import app" with the two environment variables set imports cleanly.

Step 2: Serve TwiML to open a bidirectional media stream

When a call connects, Twilio asks your webhook what to do. You answer with TwiML, Twilio’s XML instruction set. The critical instruction is <Connect><Stream>, which opens a bidirectional WebSocket: you receive the caller’s audio and send the agent’s audio back over the same socket.

1 @app.post("/twiml")
2 async def twiml(request: Request) -> Response:
3     xml = f"""<?xml version="1.0" encoding="UTF-8"?>
4 <Response>
5   <Connect>
6     <Stream url="wss://{PUBLIC_HOSTNAME}/media" />
7   </Connect>
8 </Response>"""
9     return Response(content=xml, media_type="application/xml")

The bidirectional stream makes barge-in possible. Its one-way sibling, <Start><Stream>, cannot carry the agent’s voice and therefore cannot support interruption. With a two-way channel open, the next step bridges it to Deepgram.

Verify: curl -X POST https://YOUR_HOST/twiml returns the XML above with your wss:// URL.

Step 3: Bridge the Twilio media stream to the Voice Agent

The /media WebSocket carries the whole call. Open one Deepgram Voice Agent connection for the call, start a task that relays the agent’s output back to Twilio, then forward Twilio’s events into the agent.

1 @app.websocket("/media")
2 async def media(twilio_ws: WebSocket) -> None:
3     await twilio_ws.accept()
4     stream_sid_box: dict = {}
5 
6     async with dg_client.agent.v1.connect() as agent:
7         relay_task = asyncio.create_task(agent_to_twilio(twilio_ws, agent, stream_sid_box))
8         async for raw in twilio_ws.iter_text():
9             msg = json.loads(raw)
10             event = msg.get("event")
11             if event == "start":
12                 stream_sid_box["sid"] = msg["start"]["streamSid"]
13                 await agent.send_settings(AGENT_SETTINGS)        # configure + start the conversation
14             elif event == "media":
15                 await agent.send_media(base64.b64decode(msg["media"]["payload"]))
16             elif event == "stop":
17                 break
18         relay_task.cancel()

On start, capture streamSid (every message you send back to Twilio must reference it) and send the agent its settings, which kicks off the conversation. On media, decode the caller’s audio and forward the raw mulaw bytes straight into the agent with send_media. The agent now needs to know how to behave, and the settings define exactly that.

Step 4: Configure the agent

A single Settings message tells the Voice Agent everything: the audio format, which models to use for listening and speaking, the language model and its instructions, and a greeting. Requesting mulaw at 8 kHz for both input and output means the audio matches Twilio exactly, with no resampling anywhere.

1 AGENT_SETTINGS = AgentV1Settings.model_validate({
2     "type": "Settings",
3     "audio": {
4         "input":  {"encoding": "mulaw", "sample_rate": 8000},
5         "output": {"encoding": "mulaw", "sample_rate": 8000, "container": "none"},
6     },
7     "agent": {
8         "language": "en",
9         "listen": {"provider": {"type": "deepgram", "version": "v2", "model": "flux-general-en"}},
10         "think":  {"provider": {"type": "open_ai", "model": "gpt-4o-mini"}, "prompt": PROMPT},
11         "speak":  {"provider": {"type": "deepgram", "version": "v2", "model": "flux-alexis-en"}},
12         "greeting": "Hi! Thanks for calling. How can I help you today?",
13     },
14 })

The listen provider is Flux, Deepgram’s speech-to-text model built for conversational audio. Flux runs on the /v2/listen endpoint, so the provider pins "version": "v2"; drop that field and the agent falls back to the v1 (Nova) endpoint, where flux-general-en is not a valid model. Flux adds model-integrated end-of-turn detection tuned for voice agents, with Nova-3-level accuracy. Twilio’s mulaw 8 kHz audio needs no change — the agent resamples it for Flux internally.

The speak provider is Flux TTS, Deepgram’s streaming text-to-speech model, so both ends of the pipeline are Flux. Flux voices are named flux-{voice}-en and run on the /v2/speak endpoint, so this provider pins "version": "v2" as well; drop it and the agent falls back to v1 (Aura), where flux-alexis-en is not a valid voice.

The think provider is OpenAI’s gpt-4o-mini, and a focused prompt keeps replies short and speakable: one or two sentences, no markdown — exactly what a phone call needs. Note the missing API key in the think block — Deepgram manages the LLM connection on your behalf. Once you send these settings, the agent’s replies start flowing back as audio.

Choosing the language model

The think provider accepts multiple LLM providers, so you can swap the agent’s model without touching the rest of the pipeline — speech-to-text, text-to-speech, and turn-taking all keep working unchanged. Deepgram manages the connection to whichever provider you select and bills it through your Deepgram account, so you don’t add a separate provider key to the settings.

Provider	`provider.type`	Example `model`
OpenAI (used above)	`open_ai`	`gpt-4o-mini`
Anthropic	`anthropic`	`claude-sonnet-5`

Switching providers is a two-field change inside think.provider. To run Anthropic’s Claude Sonnet 5 instead of OpenAI:

1 "think": {"provider": {"type": "anthropic", "model": "claude-sonnet-5"}, "prompt": PROMPT},

An optional temperature on the provider tunes how deterministic the replies are. See Deepgram’s Voice Agent LLM models for the full list of supported providers and models.

Verify: on a connected call, the agent speaks the greeting, and ConversationText events print to the console.

Step 5: Relay the agent’s audio back to Twilio

The Voice Agent streams its responses back in two forms over the one connection: output audio arrives as raw bytes, and everything else (transcripts, status, interruption signals) arrives as typed event objects. A single receive loop handles both.

1 async def agent_to_twilio(twilio_ws, agent, stream_sid_box):
2     while True:
3         message = await agent.recv()
4         sid = stream_sid_box.get("sid")
5 
6         if isinstance(message, bytes):                       # agent output audio
7             await twilio_ws.send_text(json.dumps({
8                 "event": "media", "streamSid": sid,
9                 "media": {"payload": base64.b64encode(message).decode()},
10             }))
11         elif isinstance(message, AgentV1UserStartedSpeaking): # barge-in (Step 6)
12             ...
13         elif isinstance(message, AgentV1ConversationText):
14             print(f"[{message.role}] {message.content}")

Because the audio is already mulaw 8 kHz, each chunk drops straight into a Twilio media frame with no conversion. Driving the loop with agent.recv() keeps the audio and the control events in order on a single task, which matters for the interruption handling that comes next.

Step 6: Handle barge-in

Barge-in turns a scripted bot into a conversational agent, and the Voice Agent does the hard part for you. When the caller starts talking over the agent, Deepgram detects it, stops the agent’s turn, and sends a UserStartedSpeaking event. Your one responsibility is to flush the audio Twilio still has buffered.

1 elif isinstance(message, AgentV1UserStartedSpeaking):
2     await twilio_ws.send_text(json.dumps({"event": "clear", "streamSid": sid}))

The clear message discards whatever Twilio has queued but not yet played, so the agent falls silent the instant the caller speaks. Deepgram has already stopped generating on its side; the clear closes the gap on Twilio’s side. That single relayed signal is the whole of barge-in here — far less work than wiring it by hand. That leaves one thing to do: place a real call.

Step 7: Run and test the agent

Start the application and open a tunnel so Twilio can reach it.

$ python app.py                   # or: uvicorn app:app --port 5050 --reload
$ ngrok http 127.0.0.1:5050       # in another terminal; copy the forwarding host into .env PUBLIC_HOSTNAME

Next, connect the phone number to your webhook. In the Twilio Console, open Phone Numbers → Manage → Active numbers → [your number] → Voice Configuration, set A call comes in to a Webhook pointing at https://YOUR_HOST/twiml with method HTTP POST, and save.

Now place the call and confirm the agent end to end.

Call the number and listen for the greeting.
Ask a question and confirm you hear a spoken reply.
Talk over the agent mid-sentence and confirm the audio cuts off, then responds to your new turn.
Watch the console for [call] started and the [assistant] / [user] conversation lines.

A successful interruption test confirms the bridge and the Voice Agent are working together.

Go further with Deepgram

Once the core agent runs, several enhancements build on the Voice Agent API.

Give the agent tools. Define functions in the settings and handle FunctionCallRequest events to let the agent look things up or take actions mid-conversation.
Choose a different voice or model. Swap the speak model for another Flux voice, or change the think provider and model, for example to Anthropic’s claude-sonnet-5. See Choosing the language model in Step 4.
Inject messages mid-call. Use the agent’s inject and update messages to steer the conversation or update the prompt while the call is live.
Reserve a static ngrok domain so your PUBLIC_HOSTNAME and Twilio webhook stay constant across restarts.