Is Grep All You Need? How Agent Harnesses Reshape Agentic Search¶

🕒 Published (v1): 2026-05-14 17:58 UTC · Source: Arxiv · link

Why this paper was selected

Isolates harness contributions to agentic search; grep-sufficiency ablation study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper empirically compares lexical (grep) and dense vector retrieval across four agent harnesses—a custom harness (Chronos) and three provider-native CLI agents (Claude Code, Codex, Gemini CLI)—on a 116-question LongMemEval subset. Inline grep uniformly outperforms inline vector search, but harness choice shifts accuracy by margins comparable to swapping retrievers, demonstrating that "retrieval" in agentic systems is inseparable from orchestration. Tool-result delivery path (inline vs. file-based) further reshuffles the grep–vector ranking, sometimes dramatically.

Problem¶

Retrieval strategies for RAG are benchmarked in isolation from agent architecture, yet modern agentic loops are iterative and harness-mediated. Prior work ignores how tool-output presentation (inline vs. file-based) and harness class (custom vs. provider-native CLI) interact with retrieval choice. Robustness to corpus noise as irrelevant sessions accumulate is also poorly characterized end-to-end.

Method¶

Two experiments on a 116-question subset of LongMemEval, which tests long-horizon conversational memory QA across six categories (knowledge-update, multi-session, single-session-user, single-session-preference, single-session-assistant, temporal-reasoning).

Experiment 1 (factorial): Retrieval mode (grep-only vs. vector-only) × tool-calling method (inline vs. programmatic/file-based) × harness (Chronos, Claude Code, Codex CLI, Gemini CLI) × model (Claude Opus 4.6, Claude Haiku 4.5, GPT-5.4, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite). Chronos is a custom LangChain harness with dynamic category-conditioned prompting and a top-15 vector pre-context block; provider CLIs execute bash-callable wrapper scripts for each retrieval tool. Inline mode injects stdout into context; programmatic mode writes results to disk and requires the agent to read them explicitly.

Experiment 2 (noise scaling): Sweeps a discrete grid of session-limit settings, progressively adding distractor sessions to the per-question haystack while holding tool-delivery fixed, to characterize retrieval degradation curves.

Evaluation uses GPT-4o as an auxiliary LLM grader with category-conditioned rubrics; accuracy = fraction of binary-affirmative judgments.

Key Contributions¶

Empirical evidence that inline grep uniformly outperforms inline vector across all 10 harness–model pairs in Experiment 1.
Demonstration that harness choice shifts accuracy by margins comparable to swapping retrievers within a fixed harness (e.g., Claude Opus 4.6 reaches 93.1% on Chronos vs. 76.7% on Claude Code with the same inline grep tool).
Characterization of programmatic (file-based) delivery as a retrieval–orchestration stress test: it can invert the grep–vector ranking and cause extreme regressions (GPT-5.4/Codex drops from 93.1% inline grep to 55.2% programmatic grep).
Analysis of backbone-strength dependence: weaker models (Claude Haiku 4.5) show larger inline grep–vector gaps, suggesting "default to vector" recommendations should be conditioned on model capability and task type.
Noise-scaling characterization showing divergent degradation curves for grep vs. vector as distractor sessions increase.

Results¶

Inline grep vs. inline vector: grep exceeds vector on all 10 harness–model pairs; largest margin Chronos + Gemini 3.1 Flash-Lite (86.2% vs. 62.9%), narrowest Claude Code + Claude Opus 4.6 (76.7% vs. 75.0%).
Chronos inline grep range: 83.6–93.1% across backbones; inline vector 62.9–83.6%.
Harness effect holding retrieval fixed: Claude Opus 4.6 + inline grep: 93.1% (Chronos) vs. 76.7% (Claude Code) — ~16 pp swing from harness alone.
Programmatic delivery inversions: programmatic vector exceeds programmatic grep on 5 of 10 pairs.
Worst programmatic regression: GPT-5.4/Codex falls from 93.1% (inline grep) to 55.2% (programmatic grep); programmatic vector recovers partially to 67.2%.
Claude Haiku 4.5 / Claude Code: 55.2% inline grep vs. 44.0% inline vector; programmatic grep collapses to 37.1%.
Tied top performance: GPT-5.4 on Codex CLI (93.1% inline grep) matches Claude Opus 4.6 on Chronos (93.1% inline grep).

Limitations¶

Experiment 2 results are truncated in the provided text; full noise-scaling curves and conclusions are not reported here.
LongMemEval rewards literal span recovery, which may systematically favor lexical search; generalization to tasks requiring semantic blending is untested.
Provider-native harness internals (context engineering, tool ergonomics, sandboxing) are opaque, so observed harness effects cannot be causally decomposed.
No hybrid retrieval condition; the grep–vector comparison is ablative, not exhaustive of the design space.
Single benchmark corpus; corpus characteristics (structured temporal events via Chronos preprocessing) are not representative of unstructured or domain-specific corpora.
116-question subset may introduce sampling variance; statistical significance is not reported.

Relevance to Harnesses / Meta-Harnesses¶

This paper is directly about harness architecture as a first-class experimental variable, establishing that the orchestration layer—prompt construction, tool surface, result routing, iteration policy—contributes accuracy variance comparable to retrieval algorithm choice. The distinction between custom harnesses (fine-grained control, dynamic prompting, explicit context management) and provider-native CLI harnesses (opaque internals, shell-native tooling) maps precisely onto the design space that meta-harness researchers must navigate when composing or evaluating agent pipelines. The programmatic delivery findings are especially relevant: they show that file-based result routing, a common pattern in meta-harness designs for context pressure management, is itself a compositional competence test that can dominate retrieval quality as an accuracy determinant.