Multilingual Code-Switching

languagestring. Option:multi

The Multilingual Code-Switching feature in Deepgram's API enables the transcription of audio that contains or switches between multiple languages. This guide will walk you through enabling this feature, how to use it with cURL, and how to analyze and interpret the response.

📘
Multi-Lingual Code Switching is only available when using the Nova-2 model.

1. Enable Feature

To enable Multilingual Code-Switching, use the following language parameter in the query string when you call Deepgram’s /listen endpoint :

language=multi

Pre-Recorded Audio

To transcribe audio from a file on your computer that contains multiple languages, run the following cURL command in a terminal or your favorite API client.

curl \
  --request POST \
  --header 'Authorization: Token YOUR_DEEPGRAM_API_KEY' \
  --header 'Content-Type: audio/wav' \
  --data-binary @youraudio.wav \
  --url 'https://api.deepgram.com/v1/listen?language=multi&model=nova-2

🚧
Replace YOUR_DEEPGRAM_API_KEY with your Deepgram API Key.

Streaming Audio

To transcribe an audio stream, initiate a websocket connection, including the parameter language=multi. For instance:

📘
We recommend using an endpointing value of 100 ms for code-switching, endpointing=100.

wss://api.deepgram.com/v1/listen?language=multi&model=nova-2&sample_rate=44100&encoding=linear16&endpointing=100

3. Analyze Response

Pre-Recorded Audio

When the file is finished processing, you’ll receive a JSON response that has the following basic structure:


{
    "metadata": {
        "transaction_key": "deprecated",
        "request_id": "2479c8c8-8285-40ac-9ab6-f0874449f793",
        "sha256": "154e291ecfa8be6ab8343560bcc109001fa7853eb537253be8e4defc9b504c33",
        "created": "2024-06-26T19:56:16.180Z",
        "duration": 1.6,
        "channels": 1,
        "models": [
            "dc8a3fe5-a395-4b75-a8b1-71c9a5a87526"
        ],
        "model_info": {
            "dc8a3fe5-a395-4b75-a8b1-71c9a5a87526": {
                "name": "2-general-nova",
                "version": "1999-06-13.21385",
                "arch": "nova-2"
            }
        }
    },
    "results": {
        "channels": [
            {
                "alternatives": [
                    {
                        "transcript": "No recuerdo mi bank password.",
                        "confidence": 0.99902344,
                        "languages": [
                            "en",
                            "es"
                        ],
                        "words": [
                            {
                                "word": "no",
                                "start": 0.08,
                                "end": 0.32,
                                "confidence": 0.9975586,
                                "language": "es"
                            },
                            {
                                "word": "recuerdo",
                                "start": 0.32,
                                "end": 0.79999995,
                                "confidence": 0.9921875,
                                "language": "es"
                            },
                            {
                                "word": "mi",
                                "start": 0.79999995,
                                "end": 1.04,
                                "confidence": 0.96777344,
                                "language": "es"
                            },
                            {
                                "word": "bank",
                                "start": 1.04,
                                "end": 1.28,
                                "confidence": 1,
                                "language": "en"
                            },
                            {
                                "word": "password",
                                "start": 1.28,
                                "end": 1.5999999,
                                "confidence": 0.9926758,
                                "language": "en"
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

In this response, we see that each channel contains:

alternatives object, which contains:
- transcript: Transcript for the audio being processed.
- confidence: Floating point value between 0 and 1 that indicates overall transcript reliability. Larger values indicate higher confidence.
- languages: Array of BCP-47 language tags for all detected languages in the channel, sorted in descending order of number of words per language.
- words: Object containing each word in the transcript, along with its start time and end time (in seconds) from the beginning of the audio, a word-level transcription confidence value, the language of the word, and the punctuated word if Smart Formatting is enabled.

Streaming Audio

When streaming audio, a Results JSON message has the following structure:

{
    "type": "Results",
    "channel_index": [
        0,
        1
    ],
    "duration": 4.0700073,
    "start": 464.47,
    "is_final": True,
    "speech_final": False,
    "channel": {
        "alternatives": [
            {
                "transcript": "será el inglés muchos",
                "confidence": 0.937473,
                "languages": [
                    "es"
                ],
                "words": [
                    {
                        "word": "será",
                        "start": 465.43,
                        "end": 465.91,
                        "confidence": 0.9494371,
                        "language": "es"
                    },
                    {
                        "word": "el",
                        "start": 465.91,
                        "end": 466.15,
                        "confidence": 0.37035784,
                        "language": "es"
                    },
                    {
                        "word": "inglés",
                        "start": 466.15,
                        "end": 466.65,
                        "confidence": 0.416623,
                        "language": "es"
                    },
                    {
                        "word": "muchos",
                        "start": 467.75,
                        "end": 468.25,
                        "confidence": 0.937473,
                        "language": "es"
                    }
                ]
            }
        ]
    },
    "metadata": {
        "request_id": "84157495-6794-4c45-b12b-95b0aeb8793f",
        "model_info": {
            "name": "2-general-nova",
            "version": "1999-05-16.19331",
            "arch": "nova-2"
        },
        "model_uuid": "cf62fbcf-2ee4-49ff-a064-d92fc81d27f4"
    },
    "from_finalize": False
}