Amazon SageMakerMonitor & secure

Health Checks & Automatic Recovery

How Deepgram containers report readiness to Amazon SageMaker through the /ping health check, and how SageMaker uses that signal to replace unhealthy instances automatically.

Amazon SageMaker polls a /ping endpoint on every instance backing your Endpoint. That response decides whether the instance receives inference requests, and whether SageMaker replaces it.

Deepgram containers report healthy only when they can actually serve inference — models loaded, inference path functional — rather than answering a static 200. An instance that has failed to load its model reports that state instead of silently collecting requests it cannot serve.

For the platform side of this contract — request timeouts, the startup window, and replacement behavior — see How Your Container Should Respond to Health Check (Ping) Requests in the AWS documentation.

While the container is starting

A Deepgram Endpoint downloads its model artifacts and loads them into GPU memory before serving anything. For larger bundles this takes several minutes. Throughout, /ping returns 503, the Endpoint stays in Creating, and SageMaker routes no traffic. The container logs the reason at INFO:

/ping returning 503 — system still initializing (expected during container startup; not yet ready to serve)

Once the models load, /ping returns 200, the Endpoint moves to InService, and SageMaker begins routing.

SageMaker allows a bounded window for a new instance to start passing health checks. Miss it and the instance launch fails, leaving the Endpoint in a failed state. Two fields on the production variant extend that window for a large bundle or a slow artifact download:

FieldPurpose
ModelDataDownloadTimeoutInSecondsTime allowed to download model artifacts from Amazon S3.
ContainerStartupHealthCheckTimeoutInSecondsTime allowed for the container to begin passing /ping health checks.

Both accept up to 3600 seconds. They are ceilings, not delays — raising them does not slow a healthy deployment.

While the Endpoint is serving

SageMaker keeps polling /ping every few seconds. If the container reaches a state where it should not serve — the inference engine stops responding, for example — it reports unhealthy and refuses new requests, so callers get a fast error instead of a stalled one.

SageMaker then replaces the instance automatically. Replacement is not instantaneous: AWS requires a sustained failure signal, so a brief blip does not cycle your fleet.

Streaming connections use a separate check

/ping reports instance health. Each bidirectional streaming connection has its own liveness check defined by the WebSocket protocol (RFC 6455): SageMaker sends a Ping frame about once a minute, the container replies with a Pong, and several consecutive unanswered Pings close that connection.

The two are independent — a closed connection does not mean the instance is unhealthy, so client applications should reconnect on an unexpected close. See Container Contract to Support Bidirectional Streaming Capabilities.

These appear in the Endpoint’s CloudWatch Log Group, /aws/sagemaker/Endpoints/YOUR_ENDPOINT_NAME. See Observability for Amazon SageMaker for how to read and filter them.

Log patternMeaning
/ping returning 503 — system still initializingLogged at INFO. Normal during startup while models load. It stops once the container is ready.
INFO Deepgram Engine is readyThe inference engine has loaded models and is accepting requests.
composite health check failedNot expected in normal operation. Contact your Deepgram representative to help troubleshoot. Include your Endpoint name, AWS Region, and the surrounding log lines.

When to take action

What you observeWhat to do
Endpoint stays in Creating, then FailedCheck the container logs — the reason is in the container output, not the SageMaker console error. If the container was still loading when the window expired, raise the two timeout fields above.
An instance was replaced once and service recoveredNothing. Automatic recovery worked.
Instances are replaced repeatedly, or composite health check failed appearsContact your Deepgram representative with the Endpoint name, Region, and log excerpts.

To catch this before your users do, alarm on Invocation5XXErrors — see Configure CloudWatch alarms.