Cross-region inference routing
Route pre-recorded and asynchronous requests across Deepgram SageMaker Endpoints in more than one AWS Region or account through one SageMaker AI inference router.
Amazon SageMaker AI cross-Region inference routing lets your application call one inference router instead of one Deepgram SageMaker Endpoint. The router sends each request to a healthy Deepgram Endpoint in its own AWS Region first. When that endpoint throttles or returns server errors, for example because it can’t add GPU instances during a traffic spike, the router shifts new requests to your Deepgram Endpoints in other Regions or AWS accounts, then shifts them back gradually after the local endpoint recovers.
The router doesn’t change your endpoints, models, containers, or auto-scaling. You still deploy and size a Deepgram Endpoint in each Region. Cross-Region inference routing costs nothing extra: you pay standard SageMaker AI and Deepgram license charges for the endpoints that serve requests, and standard Amazon CloudWatch rates for router metrics.
Supported workloads
Asynchronous endpoints are temporarily unavailable for Marketplace-hosted Deepgram. Asynchronous routing needs an asynchronous endpoint in every Region you route to. Until asynchronous endpoints are available again, use a real-time routing target for pre-recorded audio. Contact a Deepgram representative if your use case needs asynchronous processing.
Real-time routing is available in 32 commercial AWS Regions and asynchronous routing in 25. See Tools and Regions in the AWS documentation for the current list.
Before you begin
- A Deepgram Endpoint in each Region. Deploy the same Deepgram product and version in every Region you route to. Every destination must accept the same request and return the same response format; the router doesn’t check this. The Model Package ARN is Region-specific, so copy the ARN for each Region. See Deploy Deepgram on Amazon SageMaker.
- Quota in each Region. Each standby endpoint needs instance quota in its own Region. See Requesting SageMaker Quota.
- An AWS Marketplace subscription in each account. Each AWS account that owns a destination endpoint subscribes to the Deepgram product and pays for its own endpoint. See Subscribe on AWS Marketplace.
- An AWS CLI or SDK version that supports inference routers. You manage routers with the
sagemakerclient and invoke them with thesagemaker-inference-router-runtimeclient, notsagemaker-runtime. The SageMaker AI console doesn’t support inference routers. - IAM permissions. Callers need
sagemaker:InvokeInferenceRouteron the router andsagemaker:InvokeEndpointon every destination endpoint. For asynchronous routing, they needsagemaker:InvokeInferenceRouterAsyncandsagemaker:InvokeEndpointAsyncinstead.
Route pre-recorded audio
A real-time routing target lists Deepgram real-time endpoints. Each routing target represents one model, so create one routing target per Deepgram product.
Create the router
Save the routing target as routing-targets.json. This example routes between Deepgram Endpoints in us-west-2 and us-east-1:
Create the router in the Region where most of your callers run. That Region is the router’s local Region.
Run aws sagemaker describe-inference-router --region us-west-2 --inference-router-name deepgram-router and wait until Status is InService and every destination’s RoutingStatus is Routable.
Invoke the router
Call InvokeInferenceRouter with the router name and routing target name instead of EndpointName. The router passes the request body, ContentType, Accept, and CustomAttributes to the destination unchanged, so you pass Deepgram parameters in CustomAttributes as you do for synchronous invocation.
The router can retry a request once on another destination, and each attempt has up to 60 seconds, so set the client read timeout to at least 170 seconds and allow at most 2 attempts.
Each response names the endpoint that served the request (InvokedRoutedResourceArn) and why the router chose it (RoutingReason): local-preferred, failover, capacity-available, or only-healthy.
The router accepts request bodies up to 6 MB, compared with 25 MB for InvokeEndpoint on a Regional endpoint. Send larger pre-recorded files to a Regional endpoint directly.
Route asynchronous invocations
An asynchronous routing target lists only Deepgram asynchronous endpoints, and sets the Amazon S3 output and failure locations on the routing target instead of on each endpoint. The input, output, and failure buckets must be in the router’s Region and owned by the router’s account.
Upload the audio to Amazon S3 in the router’s Region, then call InvokeInferenceRouterAsync:
The response returns the OutputLocation and FailureLocation where the transcript or error lands. The router places each request on a local endpoint first, and moves it to another Region when the endpoint refuses it or doesn’t start it within MaxTimeInBacklogSeconds. A request doesn’t move after the Deepgram container starts processing it.
Each destination’s execution role needs s3:GetObject on the input and s3:PutObject on the output path. See Route asynchronous inference requests through an inference router for the full permissions and notification options.
Plan standby capacity
The router moves traffic only to healthy destinations and doesn’t add capacity, so size each standby endpoint for the traffic you expect it to absorb.
- Keep at least one instance in service in each real-time standby endpoint, and configure auto-scaling so it adds instances before it throttles. Slow responses that succeed don’t move traffic.
- To reserve a real-time endpoint’s capacity for callers in its own Region, set
CrossRegionTrafficConfigin its endpoint configuration. For example,MaxInvocationsPerSecondPerInstance=50,MaxCrossRegionPercentPerEndpoint=25limits routers in other Regions to 25% of 50 invocations per second perInServiceinstance. Requests over the cap fail withThrottlingException, and the router retries them on another destination.CrossRegionTrafficConfigdoesn’t apply to asynchronous endpoints. - An asynchronous standby can scale to zero. Set
MaxTimeInBacklogSecondslonger than the time the standby takes to scale out. - Combine routing with multiple instance types on each endpoint. Instance pools fall back within a Region; the router falls back across Regions.
Route to endpoints in other AWS accounts
A router can send requests to a Deepgram Endpoint that another AWS account owns. The endpoint owner attaches an endpoint policy with aws sagemaker put-endpoint-policy that allows the router’s account to invoke the endpoint only through that router, using the sagemaker:SourceInferenceRouterArn condition key. The router owner grants its callers permission to invoke the router and the shared endpoint, then adds the endpoint’s ARN as a destination. See Route requests to endpoints in other AWS accounts.
Limitations
- The router doesn’t support streaming, so live transcription and streaming text-to-speech can’t fail over through it.
- The router runs in one Region. If that Region is impaired, calls to the router fail even when destinations in other Regions are healthy.
- You can’t restrict a request to specific Regions. Add destinations only in Regions where your data may be processed.
- You can’t set routing weights or health thresholds, or place another Region ahead of the router’s Region.
- The router doesn’t support
TargetVariant,TargetModel,InferenceComponentName, orSessionId. - Real-time requests have a 6 MB body limit and 60 seconds of model time per attempt. Model execution timeout increases for your account don’t apply through the router.
- A router supports up to 20 routing targets with up to 10 destinations each.