Agent Judge: Agentic Evaluation Harness for Long-Horizon Production Agents
Summary
- • Standard LLM judges fail on long-horizon agents due to context limits, stateful actions, and evolving behavior
- • Agent Judge introduces three capabilities — Search, Verification, and Adaptation — to address evaluation gaps
- • The system runs as a multi-agent harness with reader, worker, and forked agents handling different evaluation tasks
- • Long trajectories are converted into queryable objects enabling multi-hop reasoning across tool calls and state changes
Details
Standard LLM judges are structurally inadequate for long-horizon agent evaluation
The analysis contends that giving a judge a query, final output, and rubric fails for three reasons: trajectory length exceeds context windows, stateful side effects cannot be verified, and static rubrics go stale as models and workflows evolve.
Long-horizon agents like Codex and Claude Code generate trajectories spanning millions of tokens
These agents compact context as they work, allowing extended runs far beyond what a single LLM judge context window can accommodate. Evaluation by a single-pass judge is therefore structurally impossible for these systems.
Stateful agent actions require evaluators to query external systems, not just inspect agent output
Production agents update CRM records, modify AWS configs, send emails, and open pull requests. An evaluator must verify these changes against actual source-of-truth systems to determine whether the agent succeeded.
Agent Judge converts long agent trajectories into queryable evidence objects for multi-hop evaluation
Messages, tool calls, retrieved documents, database responses, logs, retries, and state changes become navigable evidence. This enables evaluators to trace failure chains across early retrieval errors, failed retries, and later tool calls.
Agent Judge uses a multi-agent architecture: reader agents, worker agents, and forked agents
Reader agents inspect targeted evidence; worker agents parallelize search and verification tasks; forked agents pursue new investigative questions raised during an initial pass — mirroring the structure of the production agents being evaluated.
Verification capability checks actual environment state against source-of-truth systems
Rather than inferring whether a CRM update occurred from the agent's output, Agent Judge queries the source-of-truth system directly to confirm the change was reflected — closing the gap between what an agent claims to have done and what actually happened.
Insight = analysis or argument from the article; Context = background information; New Tech = new technical capability or methodology
What This Means
The article argues that as AI agents take on longer, more autonomous tasks with real-world side effects, the tooling used to evaluate them must become equally sophisticated — a single LLM pass over an agent's final output is no longer sufficient. The piece presents Agent Judge as a necessary architectural shift: treating evaluation itself as an agentic, multi-step investigation rather than a one-shot judgment. For teams shipping production agents, this analysis points toward a near-term need to invest in evaluation infrastructure that can query external systems, navigate long traces, and adapt rubrics continuously.