SageMaker AI Gains OpenAI-Compatible API for Inference Endpoints

SageMaker AI exposes /openai/v1 Chat Completions endpoint

All SageMaker AI real-time inference endpoints now include an /openai/v1 path that accepts Chat Completions requests and returns responses — including streaming — directly from the container. No configuration is needed; the feature is on by default for all endpoints and inference components.

Endpoint routing uses the endpoint name in the URL, not a separate registry

SageMaker routes incoming OpenAI-format requests based on the endpoint name embedded in the URL path, meaning any compliant OpenAI client works without additional service discovery or routing configuration.

Time-limited bearer tokens replace SigV4 for OpenAI-client authentication

AWS SigV4 signing has historically been the main barrier to using OpenAI-native clients with SageMaker. The new bearer token feature issues short-lived tokens compatible with standard Authorization headers, eliminating the need for custom signing middleware or SDK wrappers.

Feature enabled by default across all SageMaker endpoints and inference components

No opt-in or endpoint reconfiguration is required. Existing SageMaker AI deployments gain OpenAI API compatibility immediately, lowering the adoption barrier for teams already running models on the platform.

Agentic frameworks like Strands Agents and LangChain can now run entirely on private GPU infrastructure

Multi-step agent workflows that previously depended on OpenAI or other commercial endpoints can now be redirected to SageMaker-hosted models with no framework-level changes. This is relevant for enterprises with data residency requirements or teams seeking cost control over high-volume inference.

Inference components allow multiple models on one endpoint with per-model resource allocation

A single SageMaker endpoint can host distinct models — for example, Llama for general tasks, a fine-tuned Mistral for domain-specific work, and a smaller classification model — each with its own resource allocation. All are callable through the same OpenAI SDK interface, simplifying multi-model architectures.

Fine-tuned open-source models become drop-in replacements for commercial API calls

Teams that fine-tune open-source models on SageMaker can now serve them through the same interface their application already uses for OpenAI or other providers. Migration from commercial to self-hosted inference requires only a URL change, removing a common integration barrier.

Real-world adoption: Caffeine.AI replaced SigV4 signing in its LLM gateway with bearer tokens

Giorgio Piatti, AI/ML Engineer at Caffeine.AI, confirmed the bearer token feature allowed SageMaker to function as a drop-in provider in their Bifrost LLM gateway — which speaks the OpenAI Chat Completions protocol — with no changes to their Vercel AI SDK or standard OpenAI client integrations.