AWS Extends Amazon Bedrock AgentCore with Cross-Cloud Observability and Framework-Agnostic Evaluations
Summary
Updated Aug 26 — Amazon Bedrock AgentCore Evaluations launched, enabling framework-agnostic scoring of any agent emitting OpenTelemetry telemetry across all major frameworks.
- • AWS Bedrock AgentCore Observability now supports AI agents running outside AWS — including on-premises, GCP, and Azure environments
- • The solution uses AWS Distro for OpenTelemetry (ADOT) auto-instrumentation to route agent telemetry to a unified CloudWatch dashboard
- • Supports agents built with major frameworks including Strands Agents, LangGraph, and CrewAI regardless of deployment location
- • Enables centralized visibility into agent reasoning chains, tool invocations, token usage, latency, and harmful output detection across all environments
Updates
AgentCore Evaluations launched for framework-agnostic scoring of AI agent outputs via OpenTelemetry
Amazon Bedrock AgentCore Evaluations decouples evaluation from framework choice — any agent emitting OpenTelemetry telemetry can be scored regardless of which SDK, model, or architecture built it. Directly addresses fragmentation in evaluation tooling across LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK, and Strands Agents.
Evaluation service requires three span types to reconstruct and score any agent session
AgentCore Evaluations reads: (1) an invoke agent span for session boundaries and user inputs/outputs, (2) LLM call spans for model invocation details, and (3) tool call spans for tool invocations. These three roles cover the full execution chain regardless of underlying framework.
All major agent frameworks supported at launch via ADOT instrumentation routing to CloudWatch
Compatible frameworks include LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK, and Strands Agents. AWS Distro for OpenTelemetry (ADOT) routes telemetry spans to Amazon CloudWatch for evaluation processing, using the same ADOT infrastructure as AgentCore Observability.
Details
AgentCore Observability Extended to Non-AWS
AWS extends Amazon Bedrock AgentCore Observability beyond its native AgentCore runtime to support agents deployed on-premises, on GCP, on Azure, or any environment with internet access
ADOT Auto-Instrumentation Solution
The AWS Distro for OpenTelemetry (ADOT) runs in-process with the agent application, automatically patches frameworks and boto3 for Amazon Bedrock calls, captures generative AI semantic convention spans, and exports telemetry to CloudWatch via SigV4 authentication
Framework Compatibility
Compatible with Strands Agents, LangGraph, CrewAI, and other OpenTelemetry-compatible agent frameworks; ADOT captures reasoning steps, tool invocations, model calls, token usage, and latency in standardized OpenTelemetry generative AI semantic conventions
Three Setup Requirements
Cross-cloud observability requires: (1) aws-opentelemetry-distro pip package, (2) IAM credentials for CloudWatch authentication, and (3) specific OpenTelemetry environment variables to route telemetry to the AgentCore dashboard
What Telemetry Covers
Provides unified visibility into agent sessions, reasoning chains, tool invocations, model outputs, token usage for cost governance, and latency metrics; enables detection of hallucinations and harmful or off-topic responses across all deployment environments
Cross-Cloud Validation
AWS validated the solution on two non-AWS environments: a simulated on-premises server and Google Cloud Shell (GCP); both successfully reported sessions, traces, and span metrics to the AgentCore Observability dashboard within 2–3 minutes
Trade-off vs. AgentCore Runtime
Agents on AgentCore runtime get automatic observability configuration; non-AWS agents require manual ADOT setup but gain full deployment flexibility — AWS explicitly frames this as an intentional architectural choice for different use cases
Product Launch = new capability or feature | Tech Info = technical implementation detail | Infrastructure = deployment or architecture detail | Research = validation or testing result | Strategy = deliberate platform decision
What This Means
As enterprises increasingly deploy AI agents across heterogeneous infrastructure — mixing AWS, GCP, Azure, and on-premises servers — centralized observability becomes critical for responsible AI governance. AWS extending AgentCore Observability beyond its own cloud removes a key operational friction point: developers no longer need separate monitoring tools per environment. The ADOT-based approach leverages OpenTelemetry's emerging generative AI semantic conventions, positioning this as a standards-aligned solution rather than proprietary lock-in. For teams running agentic workloads at scale, this enables unified cost tracking, safety monitoring, and performance analytics across an entire distributed agent fleet from a single CloudWatch dashboard.