Amazon SageMaker AI Adds Bidirectional Streaming for Real-Time Voice Apps
Summary
- • Amazon SageMaker AI supports bidirectional streaming for real-time inference since November 2025
- • vLLM's Realtime API enables live speech-to-text via WebSocket at the /v1/realtime endpoint
- • Mistral AI's Voxtral-Mini-4B-Realtime-2602 is the reference deployable real-time speech model
- • SageMaker auto-bridges HTTP/2 client streaming to container-side WebSocket on port 8443
Details
SageMaker AI bidirectional streaming for real-time inference launched November 2025
Enables continuous simultaneous data flow between clients and model containers over a single persistent connection, resolving the latency problem inherent in traditional request-response inference for audio workloads.
vLLM Realtime API exposes a native WebSocket endpoint at /v1/realtime for streaming speech-to-text
Applies piecewise CUDA graph execution to reduce GPU kernel launch overhead, directly lowering per-token latency during live transcription. vLLM is open source, preserving full architectural control for deploying teams.
SageMaker AI bridges HTTP/2 client streaming to container-side WebSocket on port 8443 automatically
Eliminates the need for developers to build custom protocol-bridging infrastructure. Managed endpoints include auto-scaling, health checks, monitoring, and rolling updates out of the box.
Voxtral-Mini-4B-Realtime-2602 from Mistral AI is the reference deployment model for this architecture
A compact 4-billion-parameter model purpose-built for real-time speech scenarios, deployed to a SageMaker AI endpoint via a vLLM container. Demonstrates the pattern for voice agents, live captioning, contact center analytics, and accessibility tooling.
Product Launch = new platform capability, New Tech = model-serving advancement, Infrastructure = managed cloud layer, Tech Info = reference implementation detail
What This Means
For AI engineers and product builders, SageMaker AI's bidirectional streaming eliminates the need to build and maintain custom low-latency, protocol-aware streaming pipelines from scratch — a common infrastructure barrier for teams shipping production voice AI. The managed HTTP/2-to-WebSocket bridge means teams can deploy vLLM-served speech models with production-grade scaling and observability without writing custom networking code. Combined with compact, purpose-built models like Voxtral-Mini-4B, this lowers the practical barrier to shipping real-time voice features — live captioning, voice agents, contact center transcription — that require consistent low-latency performance at scale.