PostHog Releases Jeeves: Open-Source 9B Reasoning Classifier Outperforming Jev on JevBench
Summary
- • PostHog releases Jeeves, an open-source 9B reasoning classifier built on Qwen3.5
- • Jeeves outperforms Jev on JevBench (0.935 vs 0.866) and out-of-domain test sets
- • Adds chain-of-thought reasoning to Jev-style classifiers via CISPO training method
- • Drop-in Jev-compatible API supports yes/no, multiple-choice, and rating questions
Details
Jeeves: PostHog's open-source reasoning classifier
9B model based on Qwen3.5-9B with LoRA fine-tuning and a pointer head, plus a block-4 diffusion drafter. Full training code, data, and model weights released publicly via PostHog on HuggingFace.
Accuracy benchmarks vs Jev and Kev-9B across test sets
Test overall (out-of-domain+held-out): 0.889 vs Jev's 0.857 and Kev-9B's 0.822. JevBench hard tier (111 items): 0.865 vs Jev's 0.730. JevBench overall (231 items): 0.935 vs Jev's 0.866.
CISPO training enables reasoning before each decision
Jeeves is trained with SFT plus CISPO, allowing the model to think through a decision before outputting a calibrated probability. Thinking is optional per request, enabled via the max_think parameter.
Inference: 0.3s without thinking, 3.3s median with thinking
Benchmarked on a single H100 in FP8 mode. Chain-of-thought length can be truncated to trade accuracy for speed. The diffusion drafter is interchangeable with custom-trained checkpoints.
Jev-compatible API enables drop-in replacement deployment
Jeeves uses Jev's SystemOne endpoint format. Supports noul (yes/no), choice (multiple-choice), and score (rating scale) question types in a single request — identical to existing Jev integrations.
Transfer benchmarks: Jeeves trails Jev on MMLU-Pro
Transfer overall (MMLU-Pro + buried state): 0.746 vs Jev's 0.800, indicating reasoning does not always improve generalization. Buried state alone: 0.759 vs Jev's 0.700 — mixed results on transfer tasks.
Source: Hacker News post / PostHog HuggingFace release; no publication date available.
What This Means
Jeeves is a notable open-source contribution to the nascent 'System One' AI agent decision model ecosystem, showing that adding reasoning to Jev-style classifiers meaningfully improves accuracy on hard tasks without sacrificing the structured API interface enterprises rely on. PostHog's release of the full training stack lowers the barrier for teams wanting to fine-tune their own reasoning classifiers for domain-specific classification pipelines. The mixed transfer results — Jeeves trails Jev on MMLU-Pro — indicate the approach has tradeoffs, but the JevBench hard tier gains suggest real value in accuracy-critical agent decision scenarios. Combined with Stanford/Nvidia's CLM-8B release the same week, the Jev-ecosystem is rapidly fragmenting into competing architectures.