Skip to content

Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool use

🕒 Published (v1): 2025-12-19 20:15 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper challenges the necessity of vector databases in RAG by showing that an LLM agent equipped only with classical keyword-search CLI tools (rga, pdfgrep) achieves over 90% of standard RAG performance on document QA benchmarks. The agentic approach is simpler to set up, cheaper to operate, and outperforms RAG on structured financial documents (FinanceBench). The key insight is that ReAct-style iterative search over raw files can substitute for dense retrieval in many practical settings.

Problem

Standard RAG requires maintaining a standing vector database with embeddings, chunking pipelines, and index infrastructure, adding cost, complexity, and latency—especially when knowledge bases change frequently. Systematic comparisons between agentic tool-use alternatives and RAG are scarce, and it is unclear how much of RAG's value comes from semantic search versus simply surfacing relevant document spans.

Method

A ReAct-based LLM agent (Claude 3 Sonnet via Amazon Bedrock, temp=0.001) is given access to a Linux shell and instructed to answer questions by iteratively running three CLI tools against a local folder of PDFs: 1. pdfmetadata.sh — enumerates files and their metadata to guide document selection. 2. rga (ripgrep-all) — regex/keyword search across PDFs with page-number output. 3. pdfgrep — PDF-specific search with page-range targeting and context lines.

The agent begins every query with metadata discovery, then issues successive keyword/regex queries, expanding or refining patterns based on intermediate observations, until it converges on a final answer (Algorithm 1). No vector index is built; all retrieval is pure lexical. The baseline is a fully-managed Amazon Bedrock Knowledge Base (Titan Text Embeddings V2, 1024-dim, 300-token fixed chunks with 20% overlap, OpenSearch Serverless index, top-5 retrieval) answered with the same Claude 3 Sonnet model. Evaluation uses RAGAS metrics (faithfulness, context recall, answer correctness) across six document corpora.

Key Contributions

  • Demonstrates that lexical agentic search achieves ≥88% average attainment across all RAGAS metrics versus semantic RAG, without any vector database.
  • Introduces a minimal three-tool harness (shell + rga + pdfgrep) that can be dropped into any environment with no embedding infrastructure.
  • Shows that on complex structured documents (FinanceBench), the agent outperforms RAG: 30.40% vs. 24.24% answer correctness (+6 pp), attributed to dynamic, multi-pass interaction with document structure.
  • Provides a side-by-side comparison with Claude 3.5 Computer Use as an alternative vector-free retrieval approach, concluding parity in accuracy but lower setup overhead for the shell-tool agent.

Results

  • Faithfulness (5 datasets): Agent avg 0.787, RAG avg 0.832; attainment 94.52%. Best: LLM Survey 99.26%; worst: Llama2Paper 88.45%.
  • Context Recall: Agent avg 0.687, RAG avg 0.782; attainment 88.05%. Best: BlockchainSolana 99.62%; worst: Llama2Paper 70.56%.
  • Answer Correctness: Agent avg 0.581, RAG avg 0.636; attainment 91.48%. Best: BlockchainSolana 99.97%; worst: PaulGrahamEssay 79.91%.
  • FinanceBench answer correctness: Agent 30.40% (3-run avg), RAG 24.24% — agent wins by ~6 pp; one run (removing J&J data) reached 39.64%.
  • Interdisciplinary free-form prose (PaulGrahamEssay) shows the largest gap (attainment ~80–88%), suggesting keyword matching struggles with conceptual/argumentative text.

Limitations

  • Performance degrades on large documents and prose-heavy corpora where semantic relatedness cannot be captured by keyword co-occurrence.
  • No handling of multimedia (tables, figures) within PDFs beyond text extraction.
  • Constrained by the LLM's context window when retrieved text segments are large.
  • Struggles with ambiguous queries where the right keyword is not obvious a priori.
  • No long-term knowledge retention; each query starts fresh.
  • External shell execution introduces latency and occasional API-call failures; reliability depends on the stability of CLI tools.
  • Privacy and access-control implications of direct filesystem access are unaddressed.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct existence proof that a minimal tool harness—a prompt template + three shell commands + a ReAct loop—can replicate the retrieval function of a full RAG pipeline, with no embedding infrastructure. For harness designers, it demonstrates how wrapping commodity CLI utilities (rga, pdfgrep) as agent tools creates a composable, low-overhead retrieval stage that can be slotted into any document-QA meta-harness. The iterative observe–search–refine loop in Algorithm 1 is a reusable harness primitive: metadata discovery → broad search → targeted page-range drill-down → answer synthesis. The FinanceBench result (agent beats RAG on structured documents) suggests that dynamic, multi-pass tool invocation can outperform static retrieval for domains where chunk boundaries misalign with answer-bearing spans—a useful design principle for meta-harnesses that orchestrate heterogeneous document types.