VAKRA: New Benchmark Tests AI Agent Reasoning Across 8,000+ APIs and 62 Domains
Summary
- • VAKRA is a new executable benchmark for AI agent reasoning across 8,000+ APIs and 62 domains — current models perform poorly
- • Unlike isolated-skill benchmarks, VAKRA tests full multi-step compositional reasoning using execution traces
- • Tasks require 3-7 step chains mixing structured API calls with unstructured document retrieval
- • Release includes dataset, public leaderboard, GitHub, and failure-mode analysis
Details
VAKRA measures compositional agent reasoning, not isolated skills
Traditional benchmarks test individual capabilities. VAKRA uses full execution traces to assess whether agents reliably complete multi-step workflows combining structured API interaction with unstructured document retrieval under natural-language tool-use constraints.
8,000+ APIs across 62 domains; tasks require 3-7 reasoning steps
Agents interact with locally hosted APIs backed by real databases. The API Chaining task category includes 2,077 test instances across 54 domains. SLOT-BIRD provides 7 generic data manipulation tools; SEL-BIRD extends this with domain-specialized operations. Tasks chain 1-12 tool calls.
MCP used for tool interaction; server-side data storage prevents bloat
The benchmark uses Model Context Protocol (MCP) for all tool interactions. A special get_data tool initializes data sources and stores full datasets server-side, preventing inefficient large data transfers over the protocol while returning a lightweight preview to the agent.
Current models perform poorly; failure-mode analysis included
All evaluated models perform poorly on VAKRA. The release goes beyond a leaderboard by including analysis of observed failure modes across task categories — giving practitioners diagnostic signal on where agent reasoning breaks down in enterprise-like conditions.
Research = findings and methodology; Tech Info = technical architecture details
What This Means
VAKRA raises the bar for AI agent evaluation — moving beyond question-answering into actual multi-step tool execution across realistic enterprise domains. The finding that current models perform poorly is a useful corrective to optimistic claims about agent readiness for complex workflows. For AI practitioners building or evaluating agentic systems, the open leaderboard and execution-trace methodology offer a more honest stress test than existing benchmarks, with failure-mode analysis pointing to where work remains.
Sources
- Evaluating Agent ReasoningHuggingface