Skip to content

SageMaker now flags latency drift and idle GPUs on its own

Short answer

Amazon SageMaker AI now monitors generative AI inference endpoints running vLLM containers on GPU instances by default. It reports latency drift, mostly idle GPUs and missed optimizations through AWS Health, then lets you benchmark alternatives before changing anything. Monitoring needs no setup and has no GPU cost.

What this means for operators

If your team hosts its own model on SageMaker, for example a support assistant or an internal drafting tool, you will start receiving AWS Health notices without having configured anything. Decide now who in the account receives them: the person who owns the endpoint and the AWS bill. A finding is a prompt to run a benchmark. Weigh the savings against the headroom you need for expected growth, and apply any change to one variant first. If a model is not meant to be touched, set the opt-out variable on its container.

Amazon has added default monitoring to SageMaker AI inference endpoints, according to a post on the AWS Machine Learning Blog. The idea is that an endpoint tuned at launch slowly drifts as traffic, prompt length and serving software change, and nobody notices because nothing actually fails.

What is monitored

Monitoring is on for endpoints that use vLLM serving containers on GPU instances. It needs no setup. SageMaker looks at operational signals only: latency, GPU utilization and the endpoint's serving configuration. AWS states it does not read requests or responses.

Three kinds of findings are produced:

  • Latency drift: median end-to-end latency or time to first token rising compared with the previous week.
  • Unused capacity: GPUs you pay for that stay mostly idle.
  • Configuration gaps: a serving container that lacks recent performance enhancements or security fixes, or an improvement like speculative decoding or GPU kernel tuning that the model is capable of using but the endpoint does not employ.

Because latency is compared with the endpoint's own history, you do not set a threshold in advance.

How findings reach you

Notices arrive through AWS Health and by email, one per endpoint, covering all of that endpoint's findings. Each states which endpoint is affected, what was observed, why it matters and a recommended next step. They are specific to an AWS Region, so you see them when signed in to the Region where the endpoint runs. AWS also caps how many an account receives at once.

The recommended action opens SageMaker Studio with the endpoint and its recent traffic already loaded. An agent summarizes the finding and asks for confirmation before acting. It then generates a notebook, with each step in its own cell, that starts an inference recommendation job. The job compares your current setup with alternatives on response time, throughput and cost per million tokens.

What it costs and what it changes

Monitoring itself carries no GPU cost. Benchmarks run on separate compute and are billed at standard SageMaker rates, or against your ML reservation if you hold one. Your production endpoint stays unchanged until you apply a result. The post suggests applying a change to a single production variant first and keeping the old configuration available for rollback.

The example scenarios in the post include a chat assistant slowed by longer retrieval-based prompts, a code-completion model that never enabled speculative decoding, and a summarization endpoint whose busiest GPU used 12 percent or less of its capacity for most of a week. In that last case, AWS notes that keeping the current size can be the right call if growth is planned.

Opting out

To exclude an endpoint, set the OPT_OUT_TRACKING environment variable, with any value, in the model container's environment when calling CreateModel. SageMaker then collects no monitoring telemetry for that endpoint, including engine metrics and GPU metrics, and sends no notifications for it.

Acting on a finding requires access to AWS Health and AWS User Notifications, a SageMaker Studio domain, IAM permissions to run recommendation jobs and update endpoints, and enough service quota for the benchmark instances.

Source: AWS Machine Learning Blog

Next step

Visibility Analyzer

An AI news item will not tell you how assistants see your own site. The Visibility Analyzer checks that on your live site: which answers cite you, which pages an engine cannot retrieve, and what to fix first. Free to run.

Run a free visibility audit

Free to run. No card.

Fee
Free
Length
One run, minutes

Free tier: two analyses a day, no card required.