Skip to navigation

Cross-region inference routing

Route pre-recorded and asynchronous requests across Deepgram SageMaker Endpoints in more than one AWS Region or account through one SageMaker AI inference router.

Amazon SageMaker AI cross-Region inference routing lets your application call one inference router instead of one Deepgram SageMaker Endpoint. The router sends each request to a healthy Deepgram Endpoint in its own AWS Region first. When that endpoint throttles or returns server errors, for example because it can’t add GPU instances during a traffic spike, the router shifts new requests to your Deepgram Endpoints in other Regions or AWS accounts, then shifts them back gradually after the local endpoint recovers.

The router doesn’t change your endpoints, models, containers, or auto-scaling. You still deploy and size a Deepgram Endpoint in each Region. Cross-Region inference routing costs nothing extra: you pay standard SageMaker AI and Deepgram license charges for the endpoints that serve requests, and standard Amazon CloudWatch rates for router metrics.

Supported workloads

WorkloadDeepgram endpointRouter APISupported
Pre-recorded (batch) audioReal-timeInvokeInferenceRouterYes, up to 6 MB per request and 60 seconds of model time
Asynchronous invocationsAsynchronousInvokeInferenceRouterAsyncYes, with input and output in Amazon S3
Streaming audioReal-time—No. The router doesn’t support streaming, so call InvokeEndpointWithBidirectionalStream on a Regional endpoint directly.

Asynchronous endpoints are temporarily unavailable for Marketplace-hosted Deepgram. Asynchronous routing needs an asynchronous endpoint in every Region you route to. Until asynchronous endpoints are available again, use a real-time routing target for pre-recorded audio. Contact a Deepgram representative if your use case needs asynchronous processing.

Real-time routing is available in 32 commercial AWS Regions and asynchronous routing in 25. See Tools and Regions in the AWS documentation for the current list.

Before you begin

  • A Deepgram Endpoint in each Region. Deploy the same Deepgram product and version in every Region you route to. Every destination must accept the same request and return the same response format; the router doesn’t check this. The Model Package ARN is Region-specific, so copy the ARN for each Region. See Deploy Deepgram on Amazon SageMaker.
  • Quota in each Region. Each standby endpoint needs instance quota in its own Region. See Requesting SageMaker Quota.
  • An AWS Marketplace subscription in each account. Each AWS account that owns a destination endpoint subscribes to the Deepgram product and pays for its own endpoint. See Subscribe on AWS Marketplace.
  • An AWS CLI or SDK version that supports inference routers. You manage routers with the sagemaker client and invoke them with the sagemaker-inference-router-runtime client, not sagemaker-runtime. The SageMaker AI console doesn’t support inference routers.
  • IAM permissions. Callers need sagemaker:InvokeInferenceRouter on the router and sagemaker:InvokeEndpoint on every destination endpoint. For asynchronous routing, they need sagemaker:InvokeInferenceRouterAsync and sagemaker:InvokeEndpointAsync instead.

Route pre-recorded audio

A real-time routing target lists Deepgram real-time endpoints. Each routing target represents one model, so create one routing target per Deepgram product.

1

Create the router

Save the routing target as routing-targets.json. This example routes between Deepgram Endpoints in us-west-2 and us-east-1:

[
{
"Name": "nova-3",
"Destinations": [
{ "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/deepgram-nova-3-usw2" },
{ "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/deepgram-nova-3-use1" }
]
}
]

Create the router in the Region where most of your callers run. That Region is the router’s local Region.

aws sagemaker create-inference-router \
--region us-west-2 \
--inference-router-name deepgram-router \
--routing-targets file://routing-targets.json

Run aws sagemaker describe-inference-router --region us-west-2 --inference-router-name deepgram-router and wait until Status is InService and every destination’s RoutingStatus is Routable.

2

Invoke the router

Call InvokeInferenceRouter with the router name and routing target name instead of EndpointName. The router passes the request body, ContentType, Accept, and CustomAttributes to the destination unchanged, so you pass Deepgram parameters in CustomAttributes as you do for synchronous invocation.

import json
import boto3
from botocore.config import Config
runtime = boto3.client(
"sagemaker-inference-router-runtime",
region_name="us-west-2",
config=Config(read_timeout=170, retries={"total_max_attempts": 2, "mode": "standard"}),
)
with open("audio.wav", "rb") as f:
response = runtime.invoke_inference_router(
InferenceRouterName="deepgram-router",
RoutingTargetName="nova-3",
ContentType="audio/wav",
Accept="application/json",
CustomAttributes="v1/listen?model=nova-3&language=en&punctuate=true",
Body=f.read(),
)
print(response["InvokedRoutedResourceArn"], response["RoutingReason"])
result = json.loads(response["Body"].read())
transcript = result["results"]["channels"][0]["alternatives"][0]["transcript"]

The router can retry a request once on another destination, and each attempt has up to 60 seconds, so set the client read timeout to at least 170 seconds and allow at most 2 attempts.

Each response names the endpoint that served the request (InvokedRoutedResourceArn) and why the router chose it (RoutingReason): local-preferred, failover, capacity-available, or only-healthy.

The router accepts request bodies up to 6 MB, compared with 25 MB for InvokeEndpoint on a Regional endpoint. Send larger pre-recorded files to a Regional endpoint directly.

Route asynchronous invocations

An asynchronous routing target lists only Deepgram asynchronous endpoints, and sets the Amazon S3 output and failure locations on the routing target instead of on each endpoint. The input, output, and failure buckets must be in the router’s Region and owned by the router’s account.

[
{
"Name": "nova-3-async",
"Destinations": [
{ "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/deepgram-nova-3-async-usw2" },
{ "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/deepgram-nova-3-async-use1" }
],
"AsyncInferenceConfig": {
"OutputConfig": {
"S3OutputPath": "s3://amzn-s3-demo-bucket/output/",
"S3FailurePath": "s3://amzn-s3-demo-bucket/failure/"
},
"ClientConfig": { "MaxTimeInBacklogSeconds": 900 }
}
}
]

Upload the audio to Amazon S3 in the router’s Region, then call InvokeInferenceRouterAsync:

aws sagemaker-inference-router-runtime invoke-inference-router-async \
--region us-west-2 \
--inference-router-name deepgram-router \
--routing-target-name nova-3-async \
--input-location s3://amzn-s3-demo-bucket/input/audio.wav \
--content-type audio/wav \
--custom-attributes "v1/listen?model=nova-3&language=en"

The response returns the OutputLocation and FailureLocation where the transcript or error lands. The router places each request on a local endpoint first, and moves it to another Region when the endpoint refuses it or doesn’t start it within MaxTimeInBacklogSeconds. A request doesn’t move after the Deepgram container starts processing it.

Each destination’s execution role needs s3:GetObject on the input and s3:PutObject on the output path. See Route asynchronous inference requests through an inference router for the full permissions and notification options.

Plan standby capacity

The router moves traffic only to healthy destinations and doesn’t add capacity, so size each standby endpoint for the traffic you expect it to absorb.

  • Keep at least one instance in service in each real-time standby endpoint, and configure auto-scaling so it adds instances before it throttles. Slow responses that succeed don’t move traffic.
  • To reserve a real-time endpoint’s capacity for callers in its own Region, set CrossRegionTrafficConfig in its endpoint configuration. For example, MaxInvocationsPerSecondPerInstance=50,MaxCrossRegionPercentPerEndpoint=25 limits routers in other Regions to 25% of 50 invocations per second per InService instance. Requests over the cap fail with ThrottlingException, and the router retries them on another destination. CrossRegionTrafficConfig doesn’t apply to asynchronous endpoints.
  • An asynchronous standby can scale to zero. Set MaxTimeInBacklogSeconds longer than the time the standby takes to scale out.
  • Combine routing with multiple instance types on each endpoint. Instance pools fall back within a Region; the router falls back across Regions.

Route to endpoints in other AWS accounts

A router can send requests to a Deepgram Endpoint that another AWS account owns. The endpoint owner attaches an endpoint policy with aws sagemaker put-endpoint-policy that allows the router’s account to invoke the endpoint only through that router, using the sagemaker:SourceInferenceRouterArn condition key. The router owner grants its callers permission to invoke the router and the shared endpoint, then adds the endpoint’s ARN as a destination. See Route requests to endpoints in other AWS accounts.

Limitations

  • The router doesn’t support streaming, so live transcription and streaming text-to-speech can’t fail over through it.
  • The router runs in one Region. If that Region is impaired, calls to the router fail even when destinations in other Regions are healthy.
  • You can’t restrict a request to specific Regions. Add destinations only in Regions where your data may be processed.
  • You can’t set routing weights or health thresholds, or place another Region ahead of the router’s Region.
  • The router doesn’t support TargetVariant, TargetModel, InferenceComponentName, or SessionId.
  • Real-time requests have a 6 MB body limit and 60 seconds of model time per attempt. Model execution timeout increases for your account don’t apply through the router.
  • A router supports up to 20 routing targets with up to 10 destinations each.