Skip to content

VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild

🕒 Published (v1): 2026-05-27 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VibeSearchBench is a 200-task bilingual benchmark that evaluates LLM agents on "VibeSearch" — multi-turn, proactive search where users progressively reveal vague intent rather than issuing complete queries upfront. Seven frontier models are evaluated under both ReAct and the OpenClaw production harness, with the best model (Claude Opus 4.6) reaching only F1 = 30.30, exposing a fundamental evaluation–experience gap in current search benchmarks.

Problem

Existing search benchmarks (BrowseComp, WideSearch, GISA, etc.) require fully-specified single-turn queries, fixed-schema outputs, and no sustained user–agent interaction. Real users start with vague intent and iteratively refine it through dialogue; benchmarks that skip this dynamic cannot diagnose how models actually perform inside deployed agent harnesses, where this is the dominant usage pattern.

Method

Each task pairs a user persona \(P\) with a schema-free ground-truth knowledge graph \(G^* = (V^*, E^*)\) whose triples \((h, r, t)\) encode the information to be retrieved. The persona defines \(K\) trigger-conditioned disclosure stages: the user simulator (LLM-backed) only reveals stage \(k\)'s requirement \(u_k\) when the agent's response satisfies a predefined trigger condition \(c_k\), forcing proactive intent elicitation. After the multi-turn interaction, the agent outputs a predicted graph \(\hat{G}\) as a triple list; evaluation computes triplet-level precision, recall, and F1 via an LLM-as-judge graph-matching framework that accepts semantic equivalence (aliases, predicate paraphrases). Models run under two harness configurations: classic ReAct and the production OpenClaw harness. 200 tasks span VibeSearch-Pro (5 professional domains) and VibeSearch-Daily (15 daily-life domains), equally split across Chinese and English.

Key Contributions

  • VibeSearchBench: first benchmark combining persona-driven progressive disclosure with schema-free graph evaluation for proactive multi-turn search
  • Progressive-disclosure user simulator: LLM-backed, trigger-conditioned; discloses needs incrementally, applies persistent pressure when conditions unmet
  • Graph-matching evaluation framework: LLM judge handles entity aliases and relation synonyms; precision/recall/F1 computed at triple level
  • Harness vs. ReAct ablation: compares OpenClaw (sub-agent collaboration, local memory, lifelong memory) against ReAct across 7 frontier models
  • Error taxonomy covering three cascading failure stages: retrieval/context management, intent elicitation, and structured knowledge construction

Results

  • Best overall F1: 30.30 (Claude Opus 4.6, OpenClaw); all 7 models score below 33
  • OpenClaw vs. ReAct: marginal overall improvement (Claude +2.43 F1, GPT-5.4 +1.88); Kimi K2.6 and Gemini-3.1 Pro show no meaningful difference
  • Proactiveness correlation: #Asst/#User ratio (agent turns per user turn) predicts performance; Claude ReAct achieves ratio 8.26 and highest F1; Gemini-3.1 Pro ratio 2.84 and lowest coverage
  • Context compression penalty: compressed trajectories drop 8–12 F1 points (0.16 vs. 0.26 average); GPT-5.4 compression rate 72% drives its F1 from 0.25 (0 compressions) to 0.12 (≥2 compressions)
  • Pro vs. Daily: Pro consistently higher F1 (Claude: 32.55 Pro vs. 28.04 Daily), except Gemini-3.1 Pro which benefits from smaller Daily graphs
  • Precision/Recall asymmetry: most models exhibit Recall > Precision (Claude: P=24.88, R=36.34); Gemini-3.1 Pro the opposite (P=34.61, R=20.63)
  • OpenClaw ablation (sub-agent, local memory, lifelong memory): none of the three mechanisms yields significant F1 improvement

Limitations

  • Only 200 tasks; task-level diversity is enforced (one topic per task) but the corpus remains small for reliable sub-domain statistics
  • User simulator uses Seed-2.0-Pro as backbone; simulator quality is bounded by that model's own capabilities and faithfulness to persona specifications
  • Evaluation via LLM-as-judge introduces its own error rates; no inter-annotator agreement reported for the judge
  • Benchmark covers only two harness configurations (ReAct and OpenClaw); generalization to other harness architectures is undemonstrated
  • Ground-truth graphs constructed by domain annotators may not fully capture all valid information, causing precision underestimation for correct but out-of-graph triples
  • No systematic study of how harness-level prompt engineering (beyond the three ablated mechanisms) affects results

Relevance to Harnesses / Meta-Harnesses

VibeSearchBench is directly situated in the harness evaluation space: it explicitly benchmarks models inside OpenClaw, a "widely adopted production harness," and compares it against the baseline ReAct framework. The key harness-relevant finding is that ablating OpenClaw's three architectural mechanisms (sub-agent collaboration, local memory, lifelong memory) yields no significant F1 gain, suggesting that current harness-level enhancements are insufficient for the VibeSearch failure modes and that advances must come at the model level. For researchers building or evaluating meta-harnesses, this provides a concrete signal that memory and decomposition scaffolding alone do not address context management and intent elicitation bottlenecks. The benchmark also operationalizes how to evaluate agents as deployed in harnesses rather than in isolated research settings, a methodological pattern applicable to any harness-oriented evaluation effort.