AWS: Continuous Evaluation Framework for Production Sales Agents on Amazon Bedrock AgentCore
Summary
- • AWS built a continuous evaluation loop for its multi-agent system serving AWS Field Sales worldwide
- • Pre-launch testing found confident but wrong answers while standard monitoring stayed green
- • Mismatched data, tool, and prompt releases dropped one test pass rate from 78% to 38%
- • Framework checks answer correctness before each release and in production, tracing failures to root cause
Details
AWS SMGS Operations built an evaluation framework for a production multi-agent system using Amazon Bedrock AgentCore
The system serves AWS Field Sales worldwide and is accessed through Amazon Quick. The framework shows whether answers are grounded and correct before every release and continuously in production.
The system uses three sub-agents: Performance Insights, Customer Insights, and Data Insights (text-to-SQL), plus dashboard access
Answers are scoped by the user's territory and role security. Users include field sales and leadership, who rely on it for Weekly Business Reviews.
Pre-launch evaluation of the Customer Insights Agent found confident but incorrect answers
Examples: it reported $410.08 as '$410.08 Million'; mislabeled Sales Revenue versus revenue excluding Adjustments; invented an ARR estimate; and presented its own calculations as sourced facts.
Operational monitoring stayed green while the agent was giving wrong answers
Per AWS, conventional health signals did not catch these quality failures; output-level evaluation was needed to surface plausible but wrong answers.
In one pre-production test, a data/tool/prompt mismatch window cut the pass rate from 78% to 38%
The drop arose from uncoordinated releases where data, tool, and prompt changes were not aligned.
Framework traces failures to root cause in data, prompt, or code
Attributing a failure to a specific layer lets teams fix the right component rather than guessing.
Product Launch = new framework, Tech Info = architecture, Research = test findings, Insight = analysis, Stat = measured figure, Strategy = approach
What This Means
AWS describes how it tests an internal AI agent system for incorrect answers that look confident, which ordinary monitoring did not catch. Agents built from several parts (data, tools, prompts) can fail sharply when those parts change out of sync. The results are self-reported by AWS.