Twilio and Deepgram

Integrate Twilio with Deepgram for real-time automatic speech recognition in Programmable Voice calls.

Solutions

To help you integrate between Twilio and Deepgram, we provide the following solutions:

• A starter server in either Python or Node. A conversation streamed to your Twilio number will be directed to our script, which will send the audio to Deepgram, and receive and print transcriptions to the screen. In a real implementation, you will likely want to provide a callback to which transcriptions can be sent.

• A Docker image (deepgram/twilio-proxy:beta) that fully integrates with our On-premises installations using the same robust Rust architecture that our other services use. For access to the Docker image, ask your Account Executive.

Starter Server

📘

The code for the starter server can be accessed in this GitHub repo.

For our starter server, we offer two scripts that can work as proxy servers to help Twilio and Deepgram share data.

• twilio-proxy-mono: Runs the proxy server for the inbound Twilio track, which represents the audio Twilio receives from the call. To learn more about Twilio tracks, see Twilio’s track documentation.

• twilio-proxy-stereo: Runs the proxy server for both the inbound and outbound Twilio tracks, which represent the audio Twilio received from the call and the audio generated by Twilio to the call. To learn more about Twilio tracks, see Twilio’s track documentation.

Before You Begin

Create a Deepgram Account and Get Your Deepgram API Key

Before you can use Deepgram products, you'll need to create a Deepgram account. After you've signed up for your free account, create a Deepgram API Key. Make note of your API Key; you will need it later.

Create a Twilio Account

Before you can use Twilio products, you'll need to create a Twilio account. In addition, if you don't currently own a Twilio phone number with Voice functionality, you'll need to purchase one.

Configure Environment

We provide sample scripts in Python and Node format and assume you have already configured a Python (3.6 or greater) or Node development environment.

Install Dependencies

For Python, we use websockets and pydub.

For Node, we use ws.

Configure the Scripts

Prior to running the scripts, you must replace the authentication with your Deepgram username and password.

12          'Authorization': 'Token YOUR_DEEPGRAM_API_KEY'

const deepgramApiKey = "INSERT_YOUR_API_KEY_HERE";

Run the Scripts

To run the scripts:

1. Clone the GitHub repo to your local machine.

2. From the command line, navigate to the cloned repository.

3. Run the script using one of the following commands:

Mono

python3 twilio-proxy-mono.py

node twilio-proxy-mono.js

Stereo

python3 twilio-proxy-stereo.py

node twilio-proxy-stereo.js

Forward Data to Proxy

Finally, you will need to forward data to the proxy scripts. You can do this by configuring Twilio to send WebSockets data to the server running the proxy scripts or by initiating a call between two people and directly forwarding the data to the proxy scripts.

Configure Twilio to Use WebSockets

See the Start Streaming Audio section of Twilio’s tutorial: "Consume a real-time Media Stream using WebSockets, Python, and Flask". In this tutorial, you will use TwiML Bins, a serverless solution that helps you provide Twilio-hosted instructions to your Twilio applications, to begin streaming your call's audio.

When calling your Twilio number, the call will be forwarded to the number you set in your TwiML Bin. The conversation will then be forked to the twilio-proxy-mono or twilio-proxy-stereo app, which will send the audio to Deepgram, receive transcriptions, and print the transcriptions to the screen. In a real implementation, you will likely want to provide a callback to which transcriptions can be sent.

Sample TwiML Bin files are as follows:

Mono

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Start>
        <Stream url="wss://my-server-address" />
     </Start>
     <Dial>my-phone-number</Dial>
</Response>

Stereo

For stereo, an additional track parameter exists.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Start>
        <Stream url="wss://my-server-address" track="both_tracks" />
     </Start>
     <Dial>my-phone-number</Dial>
</Response>

Send Call Data to Proxy Scripts

Alternatively, you can initiate a call between two people and forward the call data to the Twilio-Deepgram proxy (as seen in the Twilio + Deepgram demo) using the script in twilio/twilio-api-scripts/stream:

# twilio helper library
from twilio.rest import Client

# other imports
import time
import requests
import json
import os
import uuid

# your account sid and auth token from twilio.com/console
account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
# the twilio client
client = Client(account_sid, auth_token)
# make the outgoing call
call = client.calls.create(
  twiml = '<Response><Start><Stream url="wss://url.to.deepgram.twilio.proxy" track="both_tracks" /></Start><Dial>+11231231234</Dial></Response>', # replace number with person B, replace url
  to = '+11231231234', # person A
  from_ = '+11231231234' # your twilio number
)

// Twilio helper library
const twilio = require("twilio");

// Your account SID and auth token from twilio.com/console
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;

// The Twilio client
const client = twilio(accountSid, authToken);

// Make the outgoing call
client.calls
  .create({
    twiml:
      '<Response><Start><Stream url="wss://url.to.deepgram.twilio.proxy" track="both_tracks" /></Start><Dial>+11231231234</Dial></Response>', // replace number with person B, replace url
    to: "+11231231234", // person A
    from: "+11231231234", // your Twilio number
  })
  .then((call) => console.log(call.sid))
  .catch((err) => console.error(err));

• Be sure to replace TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN with your Twilio account information.
• Replace the url variable with the URL to the Deepgram-Twilio proxy server, and the Dial number with person B's phone number.
• Replace the to and from_ variables with person A's phone number, and your Twilio voice number, respectively.

Docker Image

We provide the Docker image deepgram/twilio-proxy:beta, which you can request from your Account Executive.

🚧

This solution is for Deepgram's On-Premises customers. Please contact us if you would like to learn more about our On-Premises solutions.

Before You Begin

Create a Deepgram Account

Before you can use Deepgram products, you'll need to create a Deepgram account.

Create a Twilio Account

Before you can use Twilio products, you'll need to create a Twilio account. In addition, if you don't currently own a Twilio phone number with Voice functionality, you'll need to purchase one.

Run the Docker Image

We recommend running the Twilio-proxy server in a Docker container configured using a Docker Compose file. We provide the following sample Compose files to show you how to do this for a Deepgram API using either our On-premise or Deepgram Hosted deployment models.

On-Premise

To deploy the Twilio proxy server to an On-Premise installation using Docker Compose, you can use the following sample Compose file.

• This sample references api.toml and engine.toml, which are used to configure an On-Premise Deepgram API and Engine. To learn more, talk to your Account Executive.

• This sample references the deepgram/actix-ws-echo:beta Docker image, which is a WebSockets echo server. You can use it to see streaming Deepgram ASR responses, or you can configure a callback URL to send Deepgram ASR responses elsewhere.

version: '2.4'

services:
  api:
    image: deepgram/onprem-api:1.32.9
    volumes:
      - '/path/to/api.toml:/api.toml:ro'
    command: -vvv serve /api.toml

  engine:
    image: deepgram/onprem-engine:3.9.1
    runtime: nvidia
    volumes:
      - '/path/to/engine.toml:/engine.toml:ro'
      - '/path/to/models:/models:ro'
    command: -v serve /engine.toml

  proxy:
    image: deepgram/twilio-proxy:beta
    ports:
      - '8080:8080'
    environment:
      - RUST_LOG=TRACE
      - PROXY_URL=0.0.0.0:8080
      - STEM_URL=ws://api:8080/v2/listen
      - CALLBACK_URL=ws://echo:8080/
    command: ''

  echo:
    image: deepgram/actix-ws-echo:beta
    environment:
      - ECHO_URL=0.0.0.0:8080
    command: ''

The Docker Compose file references the following environment variables:

Hosted

To deploy the Twilio proxy server to a Deepgram-Hosted installation using Docker Compose, you can use the following sample Compose file.

ℹ️

Note

This sample references the deepgram/actix-ws-echo:beta Docker image, which is a WebSockets echo server. You can use it to see streaming Deepgram ASR responses, or you can configure a callback URL to send Deepgram ASR responses elsewhere.

version: '2.4'

services:
  proxy:
    image: deepgram/twilio-proxy:beta
    ports:
      - '8080:8080'
    environment:
      - RUST_LOG=TRACE
      - PROXY_URL=0.0.0.0:8080
      - STEM_URL=wss://api.deepgram.com/v1/listen
      - STEM_BAUTH=YOUR_DEEPGRAM_API_KEY
      - CALLBACK_URL=ws://echo:8080/
      - CALLBACK_BAUTH=base64-encoded-callback-username:password
    command: ''

  echo:
    image: deepgram/actix-ws-echo:beta
    environment:
      - ECHO_URL=0.0.0.0:8080
    command: ''

Forward Data to Proxy

Finally, you will need to forward data to the Twilio-Deepgram proxy. You can do this by configuring Twilio to send WebSockets data to the server running the Twilio-Deepgram proxy or by initiating a call between two people and directly forwarding the data to the Twilio-Deepgram proxy.

Configure Twilio to Use WebSockets

To user the Docker Image, you must configure Twilio to forward data to the server serving the Rust program. To do this, see the Start Streaming Audio section of Twilio’s tutorial: Consume a real-time Media Stream using WebSockets, Python, and Flask. In this tutorial, you will use TwiML Bins, a serverless solution that helps you provide Twilio-hosted instructions to your Twilio applications, to begin streaming your call's audio.

When calling your Twilio number, the call will be forwarded to the number you set in your TwiML Bin. The conversation will then be forked to the Twilio-Deepgram proxy app, which will send the audio to Deepgram, receive transcriptions, and print the transcriptions to the screen. In a real implementation, you will likely want to provide a callback to which transcriptions can be sent.

Sample TwiML Bin files are as follows:

Mono

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Start>
        <Stream url="wss://my-server-address" />
     </Start>
     <Dial>my-phone-number</Dial>
</Response>

Stereo

For stereo, an additional track parameter exists.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Start>
        <Stream url="wss://my-server-address" track="both_tracks" />
     </Start>
     <Dial>my-phone-number</Dial>
</Response>

Send Call Data to Twilio-Deepgram Proxy

Alternatively, you can initiate a call between two people and forward the call data to the Twilio-Deepgram proxy (as seen in the Twilio + Deepgram demo) using the script in twilio/twilio-api-scripts/stream.py:

# twilio helper library
from twilio.rest import Client

# other imports
import time
import requests
import json
import os
import uuid

# your account sid and auth token from twilio.com/console
account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
# the twilio client
client = Client(account_sid, auth_token)
# make the outgoing call
call = client.calls.create(
  twiml = '<Response><Start><Stream url="wss://url.to.deepgram.twilio.proxy" track="both_tracks" /></Start><Dial>+11231231234</Dial></Response>', # replace number with person B, replace url
  to = '+11231231234', # person A
  from_ = '+11231231234' # your twilio number
)

• Be sure to replace TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN with your Twilio account information.
• Replace the url variable with the URL to the Deepgram-Twilio proxy server, and the Dial number with person B's phone number.
• Replace the to and from_ variables with person A's phone number, and your Twilio voice number, respectively.