detect_entities boolean. Default: false

Entity Detection

When Entity Detection is enabled, the Punctuation feature will be enabled by default.

Model Support

Entity Detection is available for both pre-recorded and streaming speech-to-text.

Enable Feature

To enable Entity Detection, when you call Deepgram’s API, add a detect_entities parameter set to true in the query string:

detect_entities=true

Pre-recorded Audio

To transcribe audio from a file on your computer, run the following curl command in a terminal or your favorite API client.

curl \
  --request POST \
  --header 'Authorization: Token YOUR_DEEPGRAM_API_KEY' \
  --header 'Content-Type: audio/wav' \
  --data-binary @youraudio.wav \
  --url 'https://api.deepgram.com/v1/listen?detect_entities=true'

Streaming Audio

To enable Entity Detection for streaming audio, establish a WebSocket connection with the detect_entities=true parameter. Remember that streaming Entity Detection is supported on Nova, Nova-2, Nova-3, and Enhanced models.

// Example filename: index.js
const { DeepgramClient } = require("@deepgram/sdk");
const fetch = require("cross-fetch");
const dotenv = require("dotenv");
dotenv.config();
// URL for the realtime streaming audio you would like to transcribe
const url = "http://stream.live.vc.bbcmedia.co.uk/bbc_world_service";
const live = async () => {
  // STEP 1: Create a Deepgram client using the API key
  const deepgram = new DeepgramClient({ apiKey: process.env.DEEPGRAM_API_KEY });
  // STEP 2: Create a live transcription connection with entity detection enabled
  const connection = await deepgram.listen.v1.connect({
    model: "nova-3",
    language: "en-US",
    smart_format: "true",
    detect_entities: "true",
  });
  // STEP 3: Listen for events from the live transcription connection
  connection.on("open", () => {
    console.log("Connection opened.");
    connection.on("close", () => {
      console.log("Connection closed.");
    });
    connection.on("message", (data) => {
      if (data.type === "Results") {
        const transcript = data.channel.alternatives[0].transcript;
        // Only process final results which contain entities
        if (data.is_final) {
          const entities = data.channel.alternatives[0].entities;
          if (entities && entities.length > 0) {
            console.log("\nTranscript:", transcript);
            console.log("Entities detected:");
            entities.forEach(entity => {
              console.log(`  - ${entity.label}: ${entity.value} (confidence: ${entity.confidence})`);
              // raw_value is present when formatting features (like smart_format) are enabled
              if (entity.raw_value) {
                console.log(`    Raw value: ${entity.raw_value}`);
              }
            });
          }
        }
      }
    });
    connection.on("error", (err) => {
      console.error("Error:", err);
    });
    // STEP 4: Fetch the audio stream and send it to the live transcription connection
    fetch(url)
      .then((r) => r.body)
      .then((res) => {
        res.on("readable", () => {
          connection.sendMedia(res.read());
        });
      });
  });
  connection.connect();
  await connection.waitForOpen();
};
live();

Analyze Response

The response structure differs between pre-recorded and streaming transcription.

Pre-recorded Response

When the file is finished processing (often after only a few seconds), you’ll receive a JSON response that has the following basic structure:

{
  "metadata": {
    "transaction_key": "string",
    "request_id": "string",
    "sha256": "string",
    "created": "string",
    "duration": 0,
    "channels": 0
  },
  "results": {
    "channels": [
      {
        "alternatives":[],
      }
    ]
  }
}

Let’s look more closely at the alternatives object:

"alternatives":[
  {
    "transcript":"Welcome to the Ai show. I'm Scott Stephenson, cofounder of Deepgram...",
    "confidence":0.9816771,
    "words": [...],
    "entities":[
      {
        "label":"NAME",
        "value":" Scott Stephenson",
        "raw_value": "scott stephenson",
        "confidence":0.9999924,
        "start_word":6,
        "end_word":8
      },
      {
        "label":"ORGANIZATION",
        "value":" Deepgram",
        "raw_value": "deepgram",
        "confidence":0.9999757,
        "start_word":10,
        "end_word":11
      },
      {
        "label": "CARDINAL",
        "value": "one",
        "raw_value": "one",
        "confidence": 1,
        "start_word": 186,
        "end_word": 187
      },
      ...
    ]
  }
]

Streaming Response

For streaming transcription, entities are included in final results only (when is_final: true). Interim results do not contain the entities array.

Here’s an example of a streaming response with Entity Detection enabled:

{
  "type": "Results",
  "channel_index": [0, 1],
  "duration": 4.64,
  "start": 0.0,
  "is_final": true,
  "speech_final": true,
  "channel": {
    "alternatives": [
      {
        "transcript": "Hi, I'm calling to update my account. My name is Jane Doe and my phone number is (555) 123-4567. You can reach me at jane.doe@email.com.",
        "confidence": 0.99,
        "words": [...],
        "entities": [
          {
            "label": "NAME",
            "value": "Jane Doe",
            "raw_value": "jane doe",
            "confidence": 0.9999,
            "start_word": 9,
            "end_word": 11
          },
          {
            "label": "PHONE_NUMBER",
            "value": "(555) 123-4567",
            "raw_value": "five five five one two three four five six seven",
            "confidence": 0.9998,
            "start_word": 15,
            "end_word": 16
          },
          {
            "label": "EMAIL_ADDRESS",
            "value": "jane.doe@email.com",
            "raw_value": "jane dot doe at email dot com",
            "confidence": 0.9999,
            "start_word": 21,
            "end_word": 22
          }
        ]
      }
    ]
  }
}

Streaming Finalization Behavior

When using Entity Detection with streaming audio, Deepgram will attempt to detect and format entities as they are spoken. For entities that seem like they may be incomplete, our system will:

• Wait until the speaker continues to non-entity speech, OR
• Finalize the transcript after 3 seconds of silence, OR
• Receive a Finalize control message
• Return only completed entities based on the available audio at that point

This approach ensures transcripts are returned promptly while maintaining entity detection precision.

Using No Delay

Setting no_delay=true forces immediate finalization of streaming transcripts without waiting for entity completion.

To use no_delay with Entity Detection:

const connection = await deepgram.listen.v1.connect({
  model: "nova-3",
  language: "en-US",
  detect_entities: "true",
  no_delay: "true"  // Forces immediate finalization, may miss entities
});

Understanding Entity Fields

Each entity object in the entities array contains the following fields:

• label: Type of entity identified (e.g., NAME, PHONE_NUMBER, EMAIL, ADDRESS).
• value: The formatted text of the entity. When Smart Formatting is enabled, this field reflects the formatted output.
• raw_value: (When formatting is enabled) The original, non-formatted text as spoken. This field is included in both pre-recorded and streaming responses when formatting features (such as Smart Formatting) are enabled.
• confidence: Floating point value between 0 and 1 that indicates overall transcript reliability. Larger values indicate higher confidence.
• start_word: Index of the first word, inclusive, of the entity in the transcript.
• end_word: Index of the last word, exclusive, of the entity in the transcript.

Key Differences Between Pre-recorded and Streaming:

Multichannel Audio

When you send multichannel audio with multichannel=true, how entity recognition uses context from other channels depends on whether the request is pre-recorded or streaming.

Pre-recorded: cross-channel context

For pre-recorded audio, entity recognition considers the full conversation across all channels. Each channel still gets its own transcript and entities array, but an entity’s label can depend on what was said on other channels.

This matters most for split-channel calls, where a question is on one channel and the answer is on another. For example, an agent on channel 0 asks for a card’s security code and expiration date, and the customer answers on channel 1:

With the agent’s question available, “O nine twenty eight” is correctly labeled as an expiration date.

Cross-channel context applies to every feature built on entity recognition, so it also affects:

• Redaction: entity-based redaction (pci, pii, phi, and specific entity types) uses the same labels. In the example above, redact=credit_card_expiration replaces the expiration date with [CREDIT_CARD_EXPIRATION_1] in the multichannel request, but leaves it unredacted when channel 1 is sent by itself.
• Smart Formatting: entity-aware formatting uses the same labels. In the example above, the expiration date formats as 09/28 in the multichannel request and as 0928 when channel 1 is sent by itself.

Only English channels supply context, because entity recognition is English-only. If you use language detection and a channel is detected as another language, that channel’s speech is not used to label entities on the other channels.

As a result, the same audio can return different entities depending on whether a channel is sent alone or together with the other channels.

Streaming: per-channel only

For streaming audio, entities on each channel are recognized using only that channel’s speech. If you need cross-channel context for entity detection or redaction on streaming calls, send the recording as a pre-recorded request after the call ends.

Identifiable Entities

View all options here: Supported Entity Types

Use Cases

Some examples of uses for Entity Detection include:

• Customers who want to improve Conversational AI and Voice Assistant by triggering particular workflows and responses based on identified name, address, location, and other key entities.
• Customers who want to enhance customer service and user experience by extracting meaningful and relevant information about key entities such as a person, organization, email, and phone number.
• Customers who want to derive meaningful and actionable insights from the audio data based on identified entities in conversations.