Amazon SageMakerMonitor & secureObservability

Prometheus & OpenTelemetry Metrics

Use SageMaker detailed observability to collect Prometheus metrics from Deepgram SageMaker Endpoints and query them with PromQL.

Amazon SageMaker detailed observability runs an AWS-managed OpenTelemetry (OTel) Collector on every instance backing your endpoint. The collector gathers per-GPU accelerator metrics, host-level system metrics, and Prometheus metrics scraped from the model container, and exports them to CloudWatch, where you query them with PromQL. This gives you finer-grained visibility than the standard CloudWatch metrics covered in Observability for Amazon SageMaker — including per-GPU utilization on multi-GPU instances and metrics emitted directly by the Deepgram engine.

This page covers turning the feature on and querying the results for Deepgram workloads. For full setup details — including account prerequisites, IAM permissions, and connecting third-party observability tools — see the AWS documentation: Detailed observability for SageMaker AI endpoints. For the standard endpoint metrics available without this feature, see Observability for Amazon SageMaker.

What you get

With detailed observability enabled, three metric sources publish to CloudWatch’s OTel-compatible metric store:

SourceExample metricsNotes
GPU (DCGM exporter)DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_MEM_COPY_UTILPer-GPU series. On multi-GPU instance types (for example Aura-2 deployments on ml.g6.12xlarge), each GPU reports separately — no more summed or averaged utilization hiding a saturated device.
Host (node exporter)node_cpu_seconds_total, node_memory_MemTotal_bytes, node_disk_io_time_seconds_totalStandard Prometheus node-exporter metrics for the instance.
Deepgram containerengine_active_requests{kind="stream"}, engine_estimated_stream_capacityThe collector scrapes the container’s Prometheus endpoint on port 8080 at /metrics. These are the same API and Engine metrics documented in the self-hosted Metrics Guide. Requires a Deepgram container version that serves this endpoint; GPU and host metrics work with every version because they are collected by AWS on the host, outside the container.

Every series carries SageMaker resource labels, including aws.sagemaker.endpoint.name, the variant name, and the instance ID, so you can filter and group across a scaled-out fleet.

These metrics work with AWS Marketplace deployments: the collector runs on the host, outside the model container, so it is unaffected by the network isolation that Marketplace model packages require.

Enable detailed observability

Detailed observability is on by default for newly created endpoints, publishing every 60 seconds — no configuration is needed to start using it on a new deployment.

To set it explicitly — for example to change the publish frequency, or to add it to an endpoint created before the feature launched — use MetricsConfig on the endpoint configuration, the same field used for enhanced metrics. The publish frequency accepts 10, 30, 60 (default), 120, 180, 240, or 300 seconds.

$aws sagemaker create-endpoint-config \
>  --endpoint-config-name YOUR_CONFIG_NAME \
>  --production-variants file://production-variants.json \
>  --metrics-config '{"EnableDetailedObservability": true, "MetricPublishFrequencyInSeconds": 60}' \
>  --region YOUR_AWS_REGION

To change the setting on an endpoint that is already serving traffic — including opting out with "EnableDetailedObservability": false — create a new endpoint configuration with the same production variants plus MetricsConfig, then run update-endpoint. The update is a blue/green deployment and the endpoint stays InService; see Update an Amazon SageMaker Endpoint.

  • EnableDetailedObservability requires a recent AWS SDK — botocore/boto3 1.43.49 or later, or an equally recent AWS CLI. Older clients reject the parameter.
  • The feature has account-level prerequisites (CloudWatch OTel-enriched metrics must be active in the account). Follow the AWS getting-started guide to enable them.

Query with PromQL

The metrics land in CloudWatch’s OTel metric store, which is queried with PromQL rather than the classic get-metric-statistics API.

CloudWatch console

In the CloudWatch console, open Metrics and use the PromQL query editor to explore the new series. Start by listing what a busy endpoint reports:

DCGM_FI_DEV_GPU_UTIL

Then filter to one endpoint. The SageMaker resource labels use OTel dotted names, so quote them in PromQL:

DCGM_FI_DEV_GPU_UTIL{"aws.sagemaker.endpoint.name"="YOUR_ENDPOINT_NAME"}

Prometheus-compatible HTTP API

CloudWatch exposes a standard Prometheus query API for these metrics at https://monitoring.YOUR_AWS_REGION.amazonaws.com/api/v1/query, authenticated with SigV4 (service name monitoring). Any tool that can sign requests works; for example with awscurl:

$awscurl --service monitoring --region YOUR_AWS_REGION \
>  "https://monitoring.YOUR_AWS_REGION.amazonaws.com/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL"

The discovery endpoints work too — list every metric name the account is receiving:

$awscurl --service monitoring --region YOUR_AWS_REGION \
>  "https://monitoring.YOUR_AWS_REGION.amazonaws.com/api/v1/label/__name__/values"

Because the API is Prometheus-compatible, you can also point Grafana or other Prometheus-native observability tools at it. See the AWS documentation for supported integrations.

Detailed-observability metrics live in the OTel metric store only — they do not appear in aws cloudwatch list-metrics or the classic metric namespaces. Use PromQL to query them.