detect_entities boolean. Default: false

Entity Detection

When Entity Detection is enabled, the Punctuation feature will be enabled by default.

Model Support

Entity Detection is available for both pre-recorded and streaming speech-to-text.

Enable Feature

To enable Entity Detection, when you call Deepgram’s API, add a detect_entities parameter set to true in the query string:

detect_entities=true

Pre-recorded Audio

To transcribe audio from a file on your computer, run the following curl command in a terminal or your favorite API client.

$
curl \
>
  --request POST \
>
  --header 'Authorization: Token YOUR_DEEPGRAM_API_KEY' \
>
  --header 'Content-Type: audio/wav' \
>
  --data-binary @youraudio.wav \
>
  --url 'https://api.deepgram.com/v1/listen?detect_entities=true'

Streaming Audio

To enable Entity Detection for streaming audio, establish a WebSocket connection with the detect_entities=true parameter. Remember that streaming Entity Detection is supported on Nova, Nova-2, Nova-3, and Enhanced models.

1
// Example filename: index.js
2
3
const { DeepgramClient } = require("@deepgram/sdk");
4
const fetch = require("cross-fetch");
5
const dotenv = require("dotenv");
6
dotenv.config();
7
8
// URL for the realtime streaming audio you would like to transcribe
9
const url = "http://stream.live.vc.bbcmedia.co.uk/bbc_world_service";
10
11
const live = async () => {
12
  // STEP 1: Create a Deepgram client using the API key
13
  const deepgram = new DeepgramClient({ apiKey: process.env.DEEPGRAM_API_KEY });
14
15
  // STEP 2: Create a live transcription connection with entity detection enabled
16
  const connection = await deepgram.listen.v1.connect({
17
    model: "nova-3",
18
    language: "en-US",
19
    smart_format: "true",
20
    detect_entities: "true",
21
  });
22
23
  // STEP 3: Listen for events from the live transcription connection
24
  connection.on("open", () => {
25
    console.log("Connection opened.");
26
27
    connection.on("close", () => {
28
      console.log("Connection closed.");
29
    });
30
31
    connection.on("message", (data) => {
32
      if (data.type === "Results") {
33
        const transcript = data.channel.alternatives[0].transcript;
34
35
        // Only process final results which contain entities
36
        if (data.is_final) {
37
          const entities = data.channel.alternatives[0].entities;
38
39
          if (entities && entities.length > 0) {
40
            console.log("\nTranscript:", transcript);
41
            console.log("Entities detected:");
42
            entities.forEach(entity => {
43
              console.log(`  - ${entity.label}: ${entity.value} (confidence: ${entity.confidence})`);
44
              // raw_value is present when formatting features (like smart_format) are enabled
45
              if (entity.raw_value) {
46
                console.log(`    Raw value: ${entity.raw_value}`);
47
              }
48
            });
49
          }
50
        }
51
      }
52
    });
53
54
    connection.on("error", (err) => {
55
      console.error("Error:", err);
56
    });
57
58
    // STEP 4: Fetch the audio stream and send it to the live transcription connection
59
    fetch(url)
60
      .then((r) => r.body)
61
      .then((res) => {
62
        res.on("readable", () => {
63
          connection.sendMedia(res.read());
64
        });
65
      });
66
  });
67
68
  connection.connect();
69
  await connection.waitForOpen();
70
};
71
72
live();

Analyze Response

The response structure differs between pre-recorded and streaming transcription.

Pre-recorded Response

When the file is finished processing (often after only a few seconds), you’ll receive a JSON response that has the following basic structure:

1
{
2
  "metadata": {
3
    "transaction_key": "string",
4
    "request_id": "string",
5
    "sha256": "string",
6
    "created": "string",
7
    "duration": 0,
8
    "channels": 0
9
  },
10
  "results": {
11
    "channels": [
12
      {
13
        "alternatives":[],
14
      }
15
    ]
16
  }
17
}

Let’s look more closely at the alternatives object:

1
"alternatives":[
2
  {
3
    "transcript":"Welcome to the Ai show. I'm Scott Stephenson, cofounder of Deepgram...",
4
    "confidence":0.9816771,
5
    "words": [...],
6
    "entities":[
7
      {
8
        "label":"NAME",
9
        "value":" Scott Stephenson",
10
        "raw_value": "scott stephenson",
11
        "confidence":0.9999924,
12
        "start_word":6,
13
        "end_word":8
14
      },
15
      {
16
        "label":"ORGANIZATION",
17
        "value":" Deepgram",
18
        "raw_value": "deepgram",
19
        "confidence":0.9999757,
20
        "start_word":10,
21
        "end_word":11
22
      },
23
      {
24
        "label": "CARDINAL",
25
        "value": "one",
26
        "raw_value": "one",
27
        "confidence": 1,
28
        "start_word": 186,
29
        "end_word": 187
30
      },
31
      ...
32
    ]
33
  }
34
]

Streaming Response

For streaming transcription, entities are included in final results only (when is_final: true). Interim results do not contain the entities array.

Here’s an example of a streaming response with Entity Detection enabled:

1
{
2
  "type": "Results",
3
  "channel_index": [0, 1],
4
  "duration": 4.64,
5
  "start": 0.0,
6
  "is_final": true,
7
  "speech_final": true,
8
  "channel": {
9
    "alternatives": [
10
      {
11
        "transcript": "Hi, I'm calling to update my account. My name is Jane Doe and my phone number is (555) 123-4567. You can reach me at jane.doe@email.com.",
12
        "confidence": 0.99,
13
        "words": [...],
14
        "entities": [
15
          {
16
            "label": "NAME",
17
            "value": "Jane Doe",
18
            "raw_value": "jane doe",
19
            "confidence": 0.9999,
20
            "start_word": 9,
21
            "end_word": 11
22
          },
23
          {
24
            "label": "PHONE_NUMBER",
25
            "value": "(555) 123-4567",
26
            "raw_value": "five five five one two three four five six seven",
27
            "confidence": 0.9998,
28
            "start_word": 15,
29
            "end_word": 16
30
          },
31
          {
32
            "label": "EMAIL_ADDRESS",
33
            "value": "jane.doe@email.com",
34
            "raw_value": "jane dot doe at email dot com",
35
            "confidence": 0.9999,
36
            "start_word": 21,
37
            "end_word": 22
38
          }
39
        ]
40
      }
41
    ]
42
  }
43
}

Streaming Finalization Behavior

When using Entity Detection with streaming audio, Deepgram will attempt to detect and format entities as they are spoken. For entities that seem like they may be incomplete, our system will:

• Wait until the speaker continues to non-entity speech, OR
• Finalize the transcript after 3 seconds of silence, OR
• Receive a Finalize control message
• Return only completed entities based on the available audio at that point

This approach ensures transcripts are returned promptly while maintaining entity detection precision.

Using No Delay

Setting no_delay=true forces immediate finalization of streaming transcripts without waiting for entity completion.

To use no_delay with Entity Detection:

1
const connection = await deepgram.listen.v1.connect({
2
  model: "nova-3",
3
  language: "en-US",
4
  detect_entities: "true",
5
  no_delay: "true"  // Forces immediate finalization, may miss entities
6
});

Understanding Entity Fields

Each entity object in the entities array contains the following fields:

• label: Type of entity identified (e.g., NAME, PHONE_NUMBER, EMAIL, ADDRESS).
• value: The formatted text of the entity. When Smart Formatting is enabled, this field reflects the formatted output.
• raw_value: (When formatting is enabled) The original, non-formatted text as spoken. This field is included in both pre-recorded and streaming responses when formatting features (such as Smart Formatting) are enabled.
• confidence: Floating point value between 0 and 1 that indicates overall transcript reliability. Larger values indicate higher confidence.
• start_word: Index of the first word, inclusive, of the entity in the transcript.
• end_word: Index of the last word, exclusive, of the entity in the transcript.

Key Differences Between Pre-recorded and Streaming:

Identifiable Entities

View all options here: Supported Entity Types

Use Cases

Some examples of uses for Entity Detection include:

• Customers who want to improve Conversational AI and Voice Assistant by triggering particular workflows and responses based on identified name, address, location, and other key entities.
• Customers who want to enhance customer service and user experience by extracting meaningful and relevant information about key entities such as a person, organization, email, and phone number.
• Customers who want to derive meaningful and actionable insights from the audio data based on identified entities in conversations.