When To Use Multichannel and Diarization
Compare Deepgram's Multichannel and Diarization features to better understand when to use each feature.
When using Deepgram's API, you have access to our Multichannel and Diarization features, which are useful in different scenarios.
Comparing Multichannel and Diarization
Multichannel and Diarization are useful features when using Deepgram's speech-to-text.
Multichannel Audio
Multichannel audio is audio that has multiple separate audio channels, and the audio in each channel is distinct.
You may have heard of stereo sound, which is sound produced from two different audio channels--one channel for the left and one channel for the right--and which causes audio to sound wider and as having more depth than mono sound. Stereo sound can be multichannel sound if the left and right channels contain different audio. This could consist of one channel for voices and one for sound effects, one channel for each person's voice (for example, in a telemedicine visit between a patient and their doctor), or one channel for multiple speakers and another channel for other speakers (for example, in a podcast where multiple interviewers are on one channel and multiple guests are on a second channel).
Multichannel sound can also have more than two channels. When recording multiple people speaking (for example, on a company-wide conference call), separating different speakers' voices into individual audio channels can make it easier to focus on one speaker when reviewing the audio file.
Diarization
Diarization is the process of separating an audio stream into segments according to speaker identity, regardless of channel. Your audio may have two speakers on one audio channel, one speaker on one audio channel and one on another, or multiple speakers on one audio channel and one speaker on multiple other channels--diarization will identify the speakers regardless of audio channel.
In short, diarization focuses on giving information about different speakers, while multichannel focuses on identifying different audio channels.
Deepgram's Multichannel Feature
You can use Deepgram's Multichannel feature by sending multichannel=true in a request via the API or an SDK. When you do so, you are telling Deepgram to transcribe each audio channel independently, and Deepgram will return a response that contains separate channels for each channel from the audio:
"channels": [
{
alternatives: [
{
transcript: "parker scarves how may i help you",
confidence: ...,
words: ...
}
]
},
{
alternatives: [
{
transcript: "i got a scarf online for my wife",
confidence: ...,
words: ...
}
]
}
]
Deepgram's Diarization Feature
You can use Deepgram's Diarization feature by sending diarize=true in a request via the API or an SDK. When you do so, you are telling Deepgram that you want to know which unique person spoke each word in the transcript, and Deepgram will return a response that identifies each word as having been spoken by a different person by labelling it with a speaker property: speaker: 0, speaker: 1, and so on.
[
{
alternatives: [
{
transcript: "parker scarves how may i help you i got a scarf online for my wife",
confidence: 0.94873047,
words: [
{
word: 'parker',
start: 1.1770647,
end: 1.4563681,
confidence: 0.7792969,
speaker: 0
},
{
word: 'scarves',
start: 1.6558706,
end: 1.8553731,
confidence: 0.5029297,
speaker: 0
},
{
word: 'how',
start: 2.0548756,
end: 2.174577,
confidence: 0.99902344,
speaker: 0
},
{
word: 'may',
start: 2.174577,
end: 2.254378,
confidence: 0.9995117,
speaker: 0
},
{
word: 'i',
start: 2.3341792,
end: 2.4538805,
confidence: 0.9980469,
speaker: 0
},
{
word: 'help',
start: 2.4538805,
end: 2.733184,
confidence: 1,
speaker: 0
},
{
word: 'you',
start: 2.733184,
end: 2.892786,
confidence: 0.9838867,
speaker: 0
},
{
word: 'i',
start: 4.089801,
end: 4.209502,
confidence: 0.54589844,
speaker: 1
},
{
word: 'got',
start: 4.209502,
end: 4.329204,
confidence: 0.6279297,
speaker: 1
},
{
word: 'a',
start: 4.329204,
end: 4.6883082,
confidence: 0.9580078,
speaker: 1
},
{
word: 'scarf',
start: 4.6883082,
end: 5.1883082,
confidence: 0.9760742,
speaker: 1
},
{
word: 'online',
start: 5.2469153,
end: 5.526219,
confidence: 0.6933594,
speaker: 1
},
{
word: 'for',
start: 5.526219,
end: 5.6459203,
confidence: 0.7602539,
speaker: 1
},
{
word: 'my',
start: 5.6459203,
end: 5.8454227,
confidence: 0.98876953,
speaker: 1
},
{
word: 'wife',
start: 5.8454227,
end: 6.044925,
confidence: 0.7709961,
speaker: 1
},
Combining Multichannel with Diarization
Combining Deepgram's Multichannel and Diarization features can provide very specific, useful information about the people speaking in multiple audio channels. For example, if your audio contains two audio channels with several people speaking on one channel and several other people speaking on the second channel, using Multichannel will allow you to split the audio by channel, while Diarization will allow you to identify the different people speaking on each channel.
Before you combine Multichannel and Diarization, it's important to understand how each feature works individually. Otherwise, you may have difficulty understanding your returned transcript.
For example, if your audio has two different people speaking, each on a different audio channel, using both Multichannel and Diarization will return two distinct transcripts for each channel with both speakers identified as the first speaker. Having both speakers identified as the first speaker may seem unusual, but it is correct--because only one person is speaking on each distinct audio channel, each person is the one speaker (speaker: 0) on their channel.
Another example: You may have an audio file that you believe is multichannel, so you expect Deepgram to return multiple different transcripts, but you receive a response that contains separate channels with identical transcripts. In this case, you may have encountered a joint stereo audio file. Sometimes, to save file space when creating or converting an audio file, multichannel audio will undergo a process that mixes the channels into one main channel. Deepgram will still identify that the audio contains two channels, but the returned transcript for each channel will be the same (all speaking parts, regardless of how many speakers the audio contains, will be combined as one transcript).
Use Cases
To really understand when to use Multichannel and when to use Diarization, let's explore some possible scenarios.
Two audio channels with the same person speaking on each channel
A person is doing a sound check to see whether sound is coming from two different inputs
transcript: "hello and welcome to the sound test we're starting from the left channel then follows right channel left channel right channel left channel right channel and once again let channel know alright thank you so much listening to me and have a nice day"
In this scenario, because the same person is speaking on both audio channels, Diarization would not be useful. However, it could be useful to break the transcript into separate audio channels using Deepgram's Multichannel feature. If you do so, you should see the following transcript returned:
"channels": [
{
alternatives: [
{
transcript: "hello and welcome to sound test we're starting from the left channel and follows left channel left channel and once again let channel know thank you so much for listening to me and have a nice day",
confidence: 0.9472656,
words: [Array]
}
]
},
{
alternatives: [
{
transcript: "hello and welcome to the sound test we're starting from there then follows right channel right channel right channel and once again right channel thank you so much and have a nice day",
confidence: 0.9326172,
words: [Array]
}
]
}
]
Two audio channels with one person on each channel
A florist is taking an order from a customer
transcript: "thank you for calling marcus flowers hello i'd like to order flowers and i think you have what i'm looking for i'd be happy to take care of your order may i have your name please",
In this scenario, because only one individual is on each channel, Diarization would not be useful (each speaker would be returned as speaker: 0 since they are on separate channels). However, it could be useful to break the transcript into separate audio channels using Deepgram's Multichannel feature. If you do so, you should see the following transcript returned:
"channels": [
{
alternatives: [
{
transcript: "thank you for calling marcus flowers i'd be happy to take care of your order may i have your name please",
confidence: 0.9819336,
words: [{
word: 'thank',
start: 0.94,
end: 1.06,
confidence: 0.99658203,
speaker: 0
},
...
]
}
]
},
{
alternatives: [
{
transcript: "hello i'd like to order flowers and i think you have what i'm looking for",
confidence: 0.9916992,
words: [{
word: 'hello',
start: 4.0095854,
end: 4.049482,
confidence: 0.9897461,
speaker: 0
},
...
]
}
]
}
]
One audio channel with two people
A news broadcast has multiple presenters
transcript: "from npr news this is all things considered i'm robert siegel and i'm michelle norris"
In this scenario, because only one audio channel exists, Multichannel will probably not provide you with enough information. However, Diarization could provide information to help you identify each person speaking. In particular, analyzing both start and end properties alongside the speaker information can help you find sections of audio where people talk over each other, which commonly occurs in natural conversation. If you use Diarization, you should see the following transcript returned:
"channels": [
{
alternatives: [
{
transcript: "from npr news this is all things considered i'm robert siegel and i'm michelle norris",
confidence: 0.9794922,
words: [
{
word: 'from',
start: 0.81824785,
end: 0.8980769,
confidence: 0.99658203,
speaker: 0
},
{
word: 'npr',
start: 1.2573076,
end: 1.3770512,
confidence: 0.95947266,
speaker: 0
},
{
word: 'news',
start: 1.4967948,
end: 1.736282,
confidence: 0.99609375,
speaker: 0
},
{
word: 'this',
start: 1.9358547,
end: 2.0555983,
confidence: 0.9897461,
speaker: 0
},
{
word: 'is',
start: 2.0555983,
end: 2.2152565,
confidence: 0.9814453,
speaker: 0
},
{
word: 'all',
start: 2.2152565,
end: 2.414829,
confidence: 0.9902344,
speaker: 0
},
{
word: 'things',
start: 2.414829,
end: 2.853889,
confidence: 0.9941406,
speaker: 0
},
{
word: 'considered',
start: 2.853889,
end: 3.2929487,
confidence: 0.9785156,
speaker: 0
},
{
word: "i'm",
start: 3.452607,
end: 3.532436,
confidence: 0.9863281,
speaker: 0
},
{
word: 'robert',
start: 3.6521795,
end: 3.8916667,
confidence: 0.98876953,
speaker: 0
},
{
word: 'siegel',
start: 4.01141,
end: 4.210983,
confidence: 0.49243164,
speaker: 0
},
{
word: 'and',
start: 4.370641,
end: 4.45047,
confidence: 0.9794922,
speaker: 1
},
{
word: "i'm",
start: 4.570214,
end: 5.049188,
confidence: 0.4260254,
speaker: 1
},
{
word: 'michelle',
start: 5.049188,
end: 5.208846,
confidence: 0.69384766,
speaker: 1
},
{
word: 'norris',
start: 5.32859,
end: 5.82859,
confidence: 0.9379883,
speaker: 1
},
Two channels with three people on one channel and one person on the other channel
In this scenario, you could combine Multichannel and Diarization to provide useful information. Here, Multichannel would separate the transcript by audio input channels, and Diarization would help you identify which person was speaking on the first channel.
Troubleshooting
Read on for explanations to some common scenarios that may seem unusual.
When using both the Multichannel and Diarization features with two people, both people are marked as the same speaker
If your audio has two different people speaking, each on a different audio channel, using both Multichannel and Diarization will return two distinct transcripts for each channel with both speakers identified as the first speaker. Having both speakers identified as the first speaker may seem unusual, but it is correct--because only one person is speaking on each distinct audio channel, each person is the one speaker (speaker: 0) on their specific channel:
"channels": [
{
alternatives: [
{
transcript: "thank you for calling marcus flowers i'd be happy to take care of your order may i have your name please",
confidence: 0.9819336,
words: [{
word: 'thank',
start: 0.94,
end: 1.06,
confidence: 0.99658203,
speaker: 0
},
...
]
}
]
},
{
alternatives: [
{
transcript: "hello i'd like to order flowers and i think you have what i'm looking for",
confidence: 0.9916992,
words: [{
word: 'hello',
start: 4.0095854,
end: 4.049482,
confidence: 0.9897461,
speaker: 0
},
...
]
}
]
}
]
When using the Multichannel feature, Deepgram returns the same transcript on each channel
Sometimes when you believe an audio file is multichannel and expect Deepgram to return multiple different transcripts, you receive a response that contains separate channels with identical transcripts. In this case, you may have encountered a joint stereo audio file. Sometimes, to save file space when creating or converting an audio file, multichannel audio will undergo a process that mixes the channels into one main channel. Deepgram will still identify that the audio contains two channels, but the returned transcript for each channel will be the same (all speaking parts, regardless of how many speakers the audio contains, will be combined as one transcript):
"channels": [
{
alternatives: [
{
transcript:
'parker scarves how may i help you i got a scarf online for my wife',
confidence: 0.9453125,
words: [Array],
},
],
},
{
alternatives: [
{
transcript:
'parker scarves how may i help you i got a scarf online for my wife',
confidence: 0.9453125,
words: [Array],
},
],
},
]
Updated 8 months ago