Open Weight LLMs Rival Closed Models on Agent Tasks at Fraction of the Cost
Summary
- • GLM-5 and MiniMax M2.7 score comparably to closed frontier models on core agent benchmarks
- • Open models cost roughly 20x less than Claude Opus 4.6 for blended production workloads
- • GLM-5 delivers 4x faster latency than Claude Opus 4.6 at 0.65s versus 2.56s
- • Evaluation covered 7 agentic categories including tool use, memory, and file operations
Details
Deep Agents evaluation finds open models viable alternatives on agent tasks
GLM-5 (z.ai) and MiniMax M2.7 scored comparably to closed frontier models across 7 agentic eval categories. Metrics included Correctness (fraction of tests solved), Solve Rate (accuracy × speed), Step Ratio, and Tool Call Ratio. Consistent performance on tool calling and instruction following was the primary finding.
MiniMax M2.7 costs $0.30/$1.20 per M tokens vs $5.00/$25.00 for Claude Opus 4.6
Pricing comparison: Claude Opus 4.6 at $5.00/$25.00, Claude Sonnet 4.6 at $3.00/$15.00, GPT-5.4 at $2.50/$15.00, GLM-5 on Baseten at $0.95/$3.15, and MiniMax M2.7 on OpenRouter at $0.30/$1.20 per million input/output tokens.
10M token/day workload costs $250/day on Claude Opus 4.6 vs $12/day on MiniMax M2.7
The ~$238/day difference compounds to roughly $87,000 annually — a meaningful cost driver for any team running high-throughput agentic applications at scale.
GLM-5 averages 0.65s latency and 70 tokens/second vs 2.56s and 34 tokens/second for Claude Opus 4.6
The 4x latency advantage and 2x throughput improvement make GLM-5 significantly more suitable for interactive products where sub-second response times are expected.
Evaluation covered 7 categories: file ops, tool use, retrieval, conversation, memory, summarization, unit tests
These categories cover the fundamentals for agentic use. Four metrics were tracked: Correctness, Solve Rate, Step Ratio, and Tool Call Ratio — providing a multidimensional view of model performance in real agentic workflows.
Inference providers Groq, Fireworks, and Baseten optimize open model serving beyond typical self-hosted capability
Managed inference providers close a key operational gap for teams considering open models, eliminating the need to build and maintain own inference infrastructure while capturing cost and latency benefits.
Open model progress on SWE-Rebench and Terminal Bench 2.0 corroborates agent parity finding
The evaluation results align with trends visible across public benchmarks. Reliable tool calling and consistent instruction following — previously differentiators for closed models — are now broadly available in open weight models.
Research = evaluation findings, Stat = quantitative data, Tech Info = methodology/technical detail, Infrastructure = deployment/serving layer, Market Impact = competitive landscape shift
What This Means
Open weight models have crossed a performance threshold where they are now credible alternatives to closed frontier models in agentic applications — not just on benchmarks but on the consistency metrics that matter for production. The cost and latency gaps are large enough to create meaningful economic incentives: teams running high-throughput workloads could save roughly $87,000 annually by switching from Claude Opus to MiniMax M2.7. For closed model providers like Anthropic and OpenAI, this increases pricing pressure in the agentic developer segment, which is among the fastest-growing use cases.
Sentiment
Broadly excited among developers and practitioners about massive cost savings and competitive performance, with some concern for closed model providers
“MiniMax M2.7 is now in AdaL. From our early access testing, the coding is strong. What stood out to us was Agent Teams... ~2.5% of Sonnet pricing. ~5% of Opus pricing”
“A massive pricing shift in AI may be closer than it looks... MiniMax M2.7: $0.30 input, $1.20 output → Claude Opus 4.6: $15.00 input, $75.00 output... 1/50th the price of Opus”
“MiniMax M2.7 pricing... $0.30/M input $1.20/M output... Claude Opus 4.6 is $15/M input. $75/M output. M2.7 is 50x cheaper than Opus on input and competing on Multi-SWE Bench”
“Chinese AI labs now produce tokens at 1/6th the cost of US labs. MiniMax charges $2-3/million tokens vs $15 for Claude. When AI agents consume 20M tokens per coding task, that cost gap becomes a structural advantage.”
“MiniMax M2.5 on Chutes costs just $0.19 input and $1.15 output... M2.5 scores 80.2% on SWE-Bench Verified while Sonnet 4.6 scores 79.6%. You might be paying 15x more... for a model that scores lower”
Split
Devs and practitioners excited about savings (~80%), analysts concerned for US closed providers (~20%)