Twilio and Deepgram TTS

Streaming audio from Deepgram Aura Text-to-Speech (TTS) into an ongoing Twilio phone call requires the use of the Twilio streaming API.

Before you Begin

Prerequisites

For the complete code used in this guide, please check out this repository.

You will need:

• A free Twilio account with a Twilio phone number.
• ngrok to let Twilio access a local server OR your own hosted server.
• Understanding of Python and using Python virtual environments.

TwiML Bin Setup

First, you will need to set up a TwiML Bin. You can refer to the docs on how to do that in the Twilio Console.

1
<?xml version="1.0" encoding="UTF-8"?>
2
3
<Response>
4
    <Say language="en">"This call may be monitored or recorded."</Say>
5
    <Connect>
6
        <Stream url="wss://a127-75-172-116-97.ngrok-free.app/twilio" />
7
    </Connect>
8
</Response>

You should replace the url with wherever you decide to deploy the server we are about to create and ensure/twilio is at the end of the url.

In the TwiML Bin example above, ngrok is used to expose the server running locally.

Using ngrok

ngrok is recommended for quick development and testing but shouldn’t be used for production instances.

To use ngrok see their documentation.

Be sure to set the port correctly to align with the server code provided by running this command when you start the ngrok server.

ngrok http 5000

Connecting a Twilio phone number

Your TwiML Bin must then be connected to one of your Twilio phone numbers so that it gets executed whenever someone calls that number. If you need to set up a phone number and connect it to your TwiML Bin, refer to the Twilio Docs.

Building the Server

Copy the twilio.pycode from the repository as we will use this in the steps below and save this code locally as with a file name of twilio.py.

At this point you’ll want to start up a virtual environment for Python. Please refer to documentation for how to do that based on your personal Python preferences.

Depending on your situation you may also need to install specific packages used in this code.

1
pip install package_name

If your TwiML Bin is setup correctly, you can now navigate to this files location in your terminal and run the server with the following command:

$
python twilio.py

OR

$
python3 twilio.py

You can then start making calls to the phone number your TwiML Bin is using. Without any further modifications, you should hear Deepgram Aura say simply: “Hello, how are you today?”

Code Tour

Let’s dive into the code used in the twilio.py file.

First, we have some import statements:

1
import asyncio
2
import base64
3
import json
4
import sys
5
import websockets
6
import ssl
7
import requests

• We are using asyncio and websockets to build an asynchronous websocket server.
• We will use base64 to handle encoding audio from Aura to pass data to Twilio.
• We will use json to deal with parsing text messages from Twilio .
• We will use requests to make HTTP requests to Deepgram’s Aura/TTS endpoint.

Next we have:

1
async def twilio_handler(twilio_ws):
2
    streamsid_queue = asyncio.Queue()

• We will be spinning up asynchronous tasks for receiving messages from, and sending messages to, Twilio.
• We will use this streamsid_queue to pass the stream sid from the twilio_receiver task to the twilio_sender task.
• We need to specify this stream sid to ensure that audio from Deepgram Aura is routed correctly to the corresponding phone call.

The twilio_receiver task is defined next:

1
    async def twilio_receiver(twilio_ws):
2
        async for message in twilio_ws:
3
            try:
4
                data = json.loads(message)
5
6
                if data['event'] == 'start':
7
                    streamsid_queue.put_nowait(data['start']['streamSid'])
8
            except:
9
                break

• This task simply loops over incoming websocket messages from Twilio and extracts the stream sid when it gets it.

Next we have the twilio_sender task:

1
    async def twilio_sender(twilio_ws):
2
        print('twilio_sender started')
3
4
        # wait to receive the streamsid for this connection from one of Twilio's messages
5
        streamsid = await streamsid_queue.get()

• We first wait to receive the stream sid.

1
        # make a Deepgram Aura TTS request specifying that we want raw mulaw audio as the output
2
        url = 'https://api.deepgram.com/v1/speak?model=aura-2-thalia-en&encoding=mulaw&sample_rate=8000&container=none'
3
        headers = {
4
            'Authorization': 'Token YOUR_DEEPGRAM_API_KEY',
5
            'Content-Type': 'application/json'
6
        }
7
        payload = {
8
            'text': 'Hello, how are you today?'
9
        }
10
        tts_response = requests.post(url, headers=headers, json=payload)

• Then we make a request to Deepgram Aura TTS to say “Hello, how are you today?” specifying an audio format of raw, 8000 Hz, mulaw.

Next we have:

1
        if tts_response.status_code == 200:
2
            raw_mulaw = tts_response.content
3
4
            # construct a Twilio media message with the raw mulaw (see https://www.twilio.com/docs/voice/twiml/stream#websocket-messages---to-twilio)
5
            media_message = {
6
                'event': 'media',
7
                'streamSid': streamsid,
8
                'media': {
9
                    'payload': base64.b64encode(raw_mulaw).decode('ascii')
10
                }
11
            }
12
13
            # send the TTS audio to the attached phonecall
14
            await twilio_ws.send(json.dumps(media_message))

• Here we package up the Deepgram Aura TTS audio in the format Twilio expects.
• We specify the stream sid.
• We send that audio back to Twilio via the websocket connection.
• To better understand what is occurring at this step please refer to the Twilio docs for more details.

Additionally, if your application requires the bot to stop speaking at any point, you can do that simply by sending a “clear” message to Twilio.

1
await twilio_ws.send(json.dumps({"event": "clear", "streamSid": streamsid}))

To close out our websocket handler, we run these two asynchronous tasks with asyncio:

1
    await asyncio.wait([
2
        asyncio.ensure_future(twilio_receiver(twilio_ws)),
3
        asyncio.ensure_future(twilio_sender(twilio_ws))
4
    ])
5
6
    await twilio_ws.close()

Finally, for some scaffolding to spin up the server and pointing requests to get handled by the above function, we have:

1
async def router(websocket, path):
2
    if path == '/twilio':
3
        print('twilio connection incoming')
4
        await twilio_handler(websocket)
5
6
def main():
7
    # use this if using ssl
8
#	ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
9
#	ssl_context.load_cert_chain('cert.pem', 'key.pem')
10
#	server = websockets.serve(router, '0.0.0.0', 443, ssl=ssl_context)
11
12
    # use this if not using ssl
13
    server = websockets.serve(router, 'localhost', 5000)
14
15
    asyncio.get_event_loop().run_until_complete(server)
16
    asyncio.get_event_loop().run_forever()
17
18
if __name__ == '__main__':
19
    sys.exit(main() or 0)

What’s Next

• Twilio and Deepgram STT