LangChain Guide: How to Start and Structure Agent Evals, From First Dataset to Production

An agent eval scores the outputs and behavior of agent runs

The guide treats the run, the full execution of an agent on a task, as the unit being scored rather than a single model response.

Agents are hard to test because outputs vary and several answers can be correct

Exact-match checks break down. A correct-looking output can still come from a wrong path, such as an unneeded tool call or an action taken without permission.

Recommended start: pick one behavior and build a dataset of 10-20 examples from real traces

The guide favors a small realistic starting set, and says to run a baseline experiment first so later changes can be compared against a known score.

Offline evals are used before shipping; online evals are used in production

Offline evals test changes against a fixed dataset. Online evals score live traffic and catch failures the dataset did not anticipate.

New production failures should be turned into dataset examples

This closes the loop between online and offline evaluation, so each observed failure becomes a regression test.

Use code evaluators where possible and LLM judges where not, with human review checking both

Code checks are deterministic for verifiable criteria. LLM judges cover subjective criteria, and human review verifies both are scoring accurately.

Multi-step agents should be evaluated at three levels: full threads, trajectories, and final state

Trajectory checks address correct answers reached via the wrong path, such as unneeded tool calls.

Benchmarks like Terminal-Bench are offline evals with longer tasks and their own environments

The guide frames public benchmarks as a kind of offline eval rather than a separate category.

Harbor runs benchmark evals and logs results to LangSmith experiments

This lets benchmark runs be tracked alongside a team's own eval experiments in the same tooling.