Deploy with Terraform | Deepgram's Docs

This guide provides a complete Terraform configuration for deploying Deepgram on Amazon SageMaker. The configuration creates an IAM execution role, a SageMaker Model from your AWS Marketplace subscription, an Endpoint Configuration, and a live Endpoint. An optional module adds auto-scaling. The configuration deploys a real-time endpoint by default; it also contains an enable_async_inference option for asynchronous endpoints, which is currently unsupported (see the warning below).

Asynchronous endpoints are temporarily not supported. Asynchronous inference (enable_async_inference = true, async_inference_config, InvokeEndpointAsync, scale-to-zero) is temporarily unavailable for Marketplace-hosted Deepgram. Leave enable_async_inference at its default of false; the async variables and resources below are documented for reference only. Need asynchronous processing? Contact a Deepgram representative.

Before running Terraform, you must subscribe to a Deepgram product on the AWS Marketplace and note the Model Package ARN. See Subscribe on AWS Marketplace for the console and Marketplace API subscription flows and how to find the Model Package ARN.

Prerequisites

Terraform 1.5 or later
The Terraform AWS provider 5.56 or later, which is the first release supporting inference_ami_version
AWS credentials configured for the target account (via environment variables, shared credentials file, or an IAM role)
An active AWS Marketplace subscription to a Deepgram SageMaker product. You can subscribe through the console or, if you provision infrastructure as code, via the Marketplace API.
The Model Package ARN for the subscribed product. See Find the Model Package ARN for how to locate it in the AWS Marketplace Manage subscriptions console.

This content moved to Subscribe on AWS Marketplace.

Find the Model Package ARN

This content moved to Subscribe on AWS Marketplace.

Project layout

deepgram-sagemaker-terraform/
├── main.tf           # Provider and resource definitions
├── variables.tf      # Input variables
├── outputs.tf        # Endpoint name, ARN, and status outputs
└── terraform.tfvars  # Your variable values (do not commit secrets)

Variables

Create variables.tf with the input variables the configuration needs. The only required value is the Model Package ARN from your Marketplace subscription.

variables.tf

variable "aws_region" {
  description = "AWS region where the SageMaker Endpoint will be deployed."
  type        = string
  default     = "us-east-1"
}
variable "model_package_arn" {
  description = "ARN of the Deepgram Model Package from AWS Marketplace."
  type        = string
}
variable "model_name" {
  description = "Name for the SageMaker Model resource."
  type        = string
  default     = "deepgram-stt"
}
variable "endpoint_name" {
  description = "Name for the SageMaker Endpoint."
  type        = string
  default     = "deepgram-stt-endpoint"
}
variable "instance_type" {
  description = "SageMaker instance type for the endpoint."
  type        = string
  default     = "ml.g6.2xlarge"
}
variable "inference_ami_version" {
  description = "SageMaker-managed host AMI (NVIDIA driver + container runtime) for the production variant. Defaults to the latest available version. Set to \"\" to use the SageMaker default for your instance type."
  type        = string
  default     = "al2023-ami-sagemaker-inference-gpu-4-1"
  validation {
    condition = contains([
      "",
      "al2-ami-sagemaker-inference-gpu-2",
      "al2-ami-sagemaker-inference-gpu-2-1",
      "al2-ami-sagemaker-inference-gpu-3-1",
      "al2023-ami-sagemaker-inference-gpu-4-1",
    ], var.inference_ami_version)
    error_message = "inference_ami_version must be a supported GPU AMI version, or \"\" to use the SageMaker default."
  }
}
variable "initial_instance_count" {
  description = "Number of instances to launch at endpoint creation."
  type        = number
  default     = 1
}
variable "variant_name" {
  description = "Name of the production variant."
  type        = string
  default     = "AllTraffic"
}
variable "deepgram_engine_env" {
  description = "Map of DEEPGRAM_ENGINE_* environment variables for TOML overrides."
  type        = map(string)
  default     = {}
}
variable "deepgram_api_env" {
  description = "Map of DEEPGRAM_API_* environment variables for TOML overrides."
  type        = map(string)
  default     = {}
}
variable "enable_autoscaling" {
  description = "Enable auto-scaling for the endpoint."
  type        = bool
  default     = false
}
variable "autoscaling_min_capacity" {
  description = "Minimum instance count for auto-scaling."
  type        = number
  default     = 1
}
variable "autoscaling_max_capacity" {
  description = "Maximum instance count for auto-scaling."
  type        = number
  default     = 4
}
variable "autoscaling_target_value" {
  description = "Target concurrent requests per instance for the scaling policy."
  type        = number
  default     = 5.0
}
variable "enable_async_inference" {
  description = "Deploy an asynchronous endpoint (queued, S3 in/out) instead of a real-time endpoint. TEMPORARILY UNSUPPORTED for Marketplace-hosted Deepgram - leave false."
  type        = bool
  default     = false
}
variable "async_s3_output_path" {
  description = "S3 URI for async transcription output, e.g. s3://my-bucket/output/. Required when enable_async_inference = true."
  type        = string
  default     = ""
  validation {
    condition     = var.async_s3_output_path == "" || can(regex("^s3://", var.async_s3_output_path))
    error_message = "async_s3_output_path must be an s3:// URI."
  }
}
variable "async_s3_failure_path" {
  description = "Optional S3 URI for async failure output, e.g. s3://my-bucket/failures/."
  type        = string
  default     = ""
}

In async mode, autoscaling_target_value is interpreted as the target ApproximateBacklogSizePerInstance (queued requests per instance), and autoscaling_min_capacity may be set to 0 to enable scale-to-zero. In real-time mode it remains concurrent-requests-per-instance with a minimum of 1.

Main configuration

Create main.tf with the provider, IAM role, and SageMaker resources. The configuration uses the Model Package ARN from your AWS Marketplace subscription to create the model without referencing a container image directly.

main.tf

###############################################################################
# Provider
###############################################################################
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.56" # inference_ami_version requires provider >= 5.56.0
    }
  }
}
provider "aws" {
  region = var.aws_region
}
###############################################################################
# IAM Role — SageMaker Execution
###############################################################################
data "aws_iam_policy_document" "sagemaker_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["sagemaker.amazonaws.com"]
    }
  }
}
resource "aws_iam_role" "sagemaker_execution" {
  name               = "${var.model_name}-execution-role"
  assume_role_policy = data.aws_iam_policy_document.sagemaker_assume_role.json
}
resource "aws_iam_role_policy_attachment" "sagemaker_full_access" {
  role       = aws_iam_role.sagemaker_execution.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
}
###############################################################################
# Async S3 access (only when enable_async_inference = true)
###############################################################################
locals {
  async_buckets = var.enable_async_inference ? toset(compact([
    try(split("/", replace(var.async_s3_output_path, "s3://", ""))[0], ""),
    try(split("/", replace(var.async_s3_failure_path, "s3://", ""))[0], ""),
  ])) : toset([])
}
resource "aws_iam_role_policy" "async_s3" {
  count = var.enable_async_inference ? 1 : 0
  name = "${var.model_name}-async-s3"
  role = aws_iam_role.sagemaker_execution.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject"]
        Resource = [for b in local.async_buckets : "arn:aws:s3:::${b}/*"]
      },
      {
        Effect   = "Allow"
        Action   = ["s3:ListBucket"]
        Resource = [for b in local.async_buckets : "arn:aws:s3:::${b}"]
      },
    ]
  })
}
###############################################################################
# Merge Deepgram environment variables
###############################################################################
locals {
  deepgram_env = merge(
    { for k, v in var.deepgram_engine_env : "DEEPGRAM_ENGINE_${k}" => v },
    { for k, v in var.deepgram_api_env : "DEEPGRAM_API_${k}" => v },
  )
}
###############################################################################
# SageMaker Model — from AWS Marketplace Model Package
###############################################################################
resource "aws_sagemaker_model" "deepgram" {
  name                     = var.model_name
  execution_role_arn       = aws_iam_role.sagemaker_execution.arn
  enable_network_isolation = true
  primary_container {
    model_package_name = var.model_package_arn
    environment        = local.deepgram_env
  }
}
###############################################################################
# SageMaker Endpoint Configuration
###############################################################################
resource "aws_sagemaker_endpoint_configuration" "deepgram" {
  name = "${var.endpoint_name}-config"
  production_variants {
    variant_name           = var.variant_name
    model_name             = aws_sagemaker_model.deepgram.name
    initial_instance_count = var.initial_instance_count
    instance_type          = var.instance_type
    inference_ami_version  = var.inference_ami_version != "" ? var.inference_ami_version : null
  }
  dynamic "async_inference_config" {
    for_each = var.enable_async_inference ? [1] : []
    content {
      output_config {
        s3_output_path  = var.async_s3_output_path
        s3_failure_path = var.async_s3_failure_path != "" ? var.async_s3_failure_path : null
      }
    }
  }
  lifecycle {
    precondition {
      condition     = !var.enable_async_inference || var.async_s3_output_path != ""
      error_message = "async_s3_output_path is required when enable_async_inference = true."
    }
  }
}
###############################################################################
# SageMaker Endpoint
###############################################################################
resource "aws_sagemaker_endpoint" "deepgram" {
  name                 = var.endpoint_name
  endpoint_config_name = aws_sagemaker_endpoint_configuration.deepgram.name
}
###############################################################################
# Auto-Scaling (optional)
###############################################################################
resource "aws_appautoscaling_target" "sagemaker" {
  count = var.enable_autoscaling ? 1 : 0
  max_capacity       = var.autoscaling_max_capacity
  min_capacity       = var.autoscaling_min_capacity
  resource_id        = "endpoint/${aws_sagemaker_endpoint.deepgram.name}/variant/${var.variant_name}"
  scalable_dimension = "sagemaker:variant:DesiredInstanceCount"
  service_namespace  = "sagemaker"
}
resource "aws_appautoscaling_policy" "sagemaker" {
  count = var.enable_autoscaling ? 1 : 0
  name               = "${var.endpoint_name}-concurrency-policy"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.sagemaker[0].resource_id
  scalable_dimension = aws_appautoscaling_target.sagemaker[0].scalable_dimension
  service_namespace  = aws_appautoscaling_target.sagemaker[0].service_namespace
  target_tracking_scaling_policy_configuration {
    target_value = var.autoscaling_target_value
    # Real-time: scale on concurrent requests per model.
    dynamic "predefined_metric_specification" {
      for_each = var.enable_async_inference ? [] : [1]
      content {
        predefined_metric_type = "SageMakerVariantConcurrentRequestsPerModelHighResolution"
      }
    }
    # Async: scale on queue depth per instance.
    dynamic "customized_metric_specification" {
      for_each = var.enable_async_inference ? [1] : []
      content {
        metric_name = "ApproximateBacklogSizePerInstance"
        namespace   = "AWS/SageMaker"
        statistic   = "Average"
        dimensions {
          name  = "EndpointName"
          value = aws_sagemaker_endpoint.deepgram.name
        }
      }
    }
    scale_in_cooldown  = 300
    scale_out_cooldown = 60
  }
}
###############################################################################
# Async scale-from-zero (only when async + autoscaling are enabled)
###############################################################################
resource "aws_appautoscaling_policy" "async_scale_from_zero" {
  count = var.enable_async_inference && var.enable_autoscaling ? 1 : 0
  name               = "${var.endpoint_name}-scale-from-zero"
  policy_type        = "StepScaling"
  resource_id        = aws_appautoscaling_target.sagemaker[0].resource_id
  scalable_dimension = aws_appautoscaling_target.sagemaker[0].scalable_dimension
  service_namespace  = aws_appautoscaling_target.sagemaker[0].service_namespace
  step_scaling_policy_configuration {
    adjustment_type         = "ChangeInCapacity"
    metric_aggregation_type = "Average"
    cooldown                = 300
    step_adjustment {
      metric_interval_lower_bound = 0
      scaling_adjustment          = 1
    }
  }
}
resource "aws_cloudwatch_metric_alarm" "async_has_backlog" {
  count = var.enable_async_inference && var.enable_autoscaling ? 1 : 0
  alarm_name          = "${var.endpoint_name}-has-backlog-without-capacity"
  namespace           = "AWS/SageMaker"
  metric_name         = "HasBacklogWithoutCapacity"
  statistic           = "Average"
  period              = 60
  evaluation_periods  = 2
  datapoints_to_alarm = 2
  threshold           = 1
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "missing"
  dimensions = {
    EndpointName = aws_sagemaker_endpoint.deepgram.name
  }
  alarm_actions = [aws_appautoscaling_policy.async_scale_from_zero[0].arn]
}

Outputs

Create outputs.tf to surface the endpoint details after terraform apply completes.

outputs.tf

output "endpoint_name" {
  description = "Name of the deployed SageMaker Endpoint."
  value       = aws_sagemaker_endpoint.deepgram.name
}
output "endpoint_arn" {
  description = "ARN of the deployed SageMaker Endpoint."
  value       = aws_sagemaker_endpoint.deepgram.arn
}
output "model_name" {
  description = "Name of the SageMaker Model."
  value       = aws_sagemaker_model.deepgram.name
}
output "execution_role_arn" {
  description = "ARN of the IAM execution role."
  value       = aws_iam_role.sagemaker_execution.arn
}
output "async_s3_output_path" {
  description = "S3 location where async transcription results are written (async mode only)."
  value       = var.enable_async_inference ? var.async_s3_output_path : null
}

Example variable values

Create a terraform.tfvars file with your specific values. Replace the model_package_arn with the ARN from your AWS Marketplace subscription.

terraform.tfvars

aws_region            = "us-east-1"
model_package_arn     = "arn:aws:sagemaker:us-east-1:123456789012:model-package/deepgram-stt-nova-3/1"
model_name            = "deepgram-streaming-stt"
endpoint_name         = "my-deepgram-stt"
instance_type         = "ml.g6.2xlarge"
inference_ami_version = "al2023-ami-sagemaker-inference-gpu-4-1"
# Optional: Deepgram configuration overrides
deepgram_engine_env = {
  "01" = "max_active_requests=120"
}
deepgram_api_env = {
  "01" = "features.listen_v2=true"
}
# Optional: Enable auto-scaling
enable_autoscaling       = true
autoscaling_min_capacity = 1
autoscaling_max_capacity = 4
autoscaling_target_value = 5.0
# Optional: deploy an asynchronous endpoint instead of real-time
# enable_async_inference   = true
# async_s3_output_path     = "s3://my-deepgram-async/output/"
# async_s3_failure_path    = "s3://my-deepgram-async/failures/"
# enable_autoscaling       = true
# autoscaling_min_capacity = 0    # async supports scale-to-zero
# autoscaling_target_value = 5.0  # target ApproximateBacklogSizePerInstance

Do not commit terraform.tfvars to version control if it contains sensitive values. Add it to .gitignore or use environment variables instead.

Deploy

Initialize the Terraform working directory

terraform init

Preview the resources Terraform will create

terraform plan

Verify the plan shows the expected resources: an IAM role, a SageMaker Model, an Endpoint Configuration, and an Endpoint.

Apply the configuration

terraform apply

Terraform creates the resources and waits for the SageMaker Endpoint to reach InService status. This typically takes several minutes.

Verify the endpoint

Confirm the endpoint is running:

aws sagemaker describe-endpoint \
  --endpoint-name $(terraform output -raw endpoint_name) \
  --region $(terraform output -raw aws_region 2>/dev/null || echo "us-east-1") \
  --query "EndpointStatus"

The output should be "InService".

Validate the endpoint

After the endpoint reaches InService, run a test inference to confirm it returns results. See Validate a Deepgram SageMaker Endpoint for the full testing guide using the dg-sagemaker test clients.

Customize the deployment

Instance types

Choose an instance type based on the Deepgram product you are deploying. GPU-accelerated instances are required.

Product	Recommended instance type	Notes
Speech-to-Text (Nova-3, Flux)	`ml.g6.2xlarge`	Single NVIDIA L4 GPU, 24 GB GPU RAM
Text-to-Speech (Aura-2)	`ml.g6.12xlarge`	4 NVIDIA L4 GPUs (Aura-2 requires 2+ GPUs)

For the full list of supported and unsupported instance types per product, including the g7 and g7e families, see Supported Products.

The host driver your instances boot with is set separately — see Inference AMI versions.

Inference AMI versions

A SageMaker Endpoint Configuration can pin an inference AMI version — the SageMaker-managed host image supplying the NVIDIA driver and container runtime your instances boot with. It is independent of the Deepgram container: it determines which driver the container runs against. If you do not set it, SageMaker selects a default for your instance type, which on older GPU families is an older driver.

AMI version	NVIDIA driver	CUDA
`al2-ami-sagemaker-inference-gpu-2`	535	12.2
`al2-ami-sagemaker-inference-gpu-2-1`	535	12.2
`al2-ami-sagemaker-inference-gpu-3-1`	550	12.4
`al2023-ami-sagemaker-inference-gpu-4-1`	580	13.0

Deepgram recommends the latest available version, al2023-ami-sagemaker-inference-gpu-4-1, which provides the NVIDIA 580 driver. Deepgram containers select the correct CUDA compatibility layer at startup based on the host driver they detect, so a newer host driver requires no change to your deployment.

Support for older driver versions may be removed in the latest Deepgram Model Package. Pin an up-to-date inference AMI version rather than relying on the SageMaker default for your instance type.

For the full list of AMI versions and their driver and CUDA versions, see InferenceAmiVersion in the SageMaker API reference. For the driver each instance family runs by default, see the SageMaker GPU driver table.

Set it through the inference_ami_version variable. Set it to an empty string to use the SageMaker default for your instance type instead.

Changing inference_ami_version on an existing deployment replaces the endpoint configuration and updates the endpoint. Expect a rolling instance replacement, not an in-place driver upgrade.

Environment variable overrides

Pass Deepgram configuration overrides through the deepgram_engine_env and deepgram_api_env variables. Each map key becomes the suffix (for example, "01", "02"), and the value is the TOML expression. See Configure Amazon SageMaker Deployments for the full reference.

deepgram_engine_env = {
  "01" = "flux.max_streams=25"
  "02" = "chunking.streaming.step=0.5"
}

VPC configuration

To deploy the endpoint inside a VPC, add a vpc_config block to the aws_sagemaker_model resource:

VPC configuration

resource "aws_sagemaker_model" "deepgram" {
  name                     = var.model_name
  execution_role_arn       = aws_iam_role.sagemaker_execution.arn
  enable_network_isolation = true
  primary_container {
    model_package_name = var.model_package_arn
    environment        = local.deepgram_env
  }
  vpc_config {
    subnets            = ["subnet-0123456789abcdef0", "subnet-0123456789abcdef1"]
    security_group_ids = ["sg-0123456789abcdef0"]
  }
}

Asynchronous endpoints

By default this configuration deploys a real-time endpoint for streaming and synchronous transcription. To instead deploy an asynchronous endpoint — for large pre-recorded files (up to 1 GB), queued processing, and scale-to-zero — set enable_async_inference = true and provide an async_s3_output_path.

Asynchronous inference is a distinct endpoint mode: an async endpoint accepts only asynchronous invocations (InvokeEndpointAsync with S3 input/output) and cannot serve streaming or synchronous requests. Switching enable_async_inference replaces the endpoint configuration and endpoint.

When async is enabled, the configuration also:

grants the execution role s3:GetObject, s3:PutObject, and s3:ListBucket on the output and failure buckets;
switches the autoscaling target metric to ApproximateBacklogSizePerInstance and allows autoscaling_min_capacity = 0 for scale-to-zero;
adds a scale-from-zero policy so the endpoint wakes on the first queued request instead of waiting for the backlog to exceed the target value.

For invocation details, see Invoke a Deepgram SageMaker Endpoint. For autoscaling details, see Auto-Scaling Asynchronous Endpoints.

Tear down

To delete all resources created by this configuration:

terraform destroy

This removes the SageMaker Endpoint, Endpoint Configuration, Model, auto-scaling resources (if enabled), and the IAM execution role. You are no longer billed for SageMaker compute after the endpoint is deleted. Your AWS Marketplace subscription remains active.

Prerequisites

Subscribe to a Deepgram product via the Marketplace API

Find the Model Package ARN

Project layout

Variables

Main configuration

Outputs

Example variable values

Deploy

Initialize the Terraform working directory

Preview the resources Terraform will create

Apply the configuration

Verify the endpoint

Validate the endpoint

Customize the deployment

Instance types

Inference AMI versions

Environment variable overrides

VPC configuration

Asynchronous endpoints

Tear down

Related resources