Skip to content

OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory

🕒 Published (v1): 2026-04-29 12:49 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Optical context retrieval encodes visual experience for long-horizon agent memory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OCR-Memory proposes storing agent interaction trajectories as images rather than text, using a VLM backbone to retrieve relevant segments via binary index selection (locate-and-transcribe) over Set-of-Mark annotated images. This sidesteps token-budget limits and hallucination-prone free-form generation, achieving state-of-the-art on Mind2Web and AppWorld under a strict 4096-token context budget.

Problem

LLM agents operating over long horizons accumulate trajectories too large to fit in a context window. Existing mitigations — summarization, experience abstraction, and text-centric compression — trade token savings for information loss, discarding structural, temporal, or procedural details critical for debugging and multi-step planning. Text retrieval via similarity search is also brittle under long-range causal dependencies.

Method

OCR-Memory maintains an external memory bank \(\mathcal{M} = \{m_i\}_{i=1}^N\) where each entry \(m_i = (I_i, \{s_{i,k}\}_{k=1}^{K_i}, \pi_i)\) stores a rendered image of a trajectory chunk, the verbatim text segments, and metadata. Retrieval proceeds in two stages:

  1. Locate: A fine-tuned DeepSeek-OCR (3B) model processes each memory image annotated with Set-of-Mark (SoM) prompting — red bounding boxes with integer IDs — and outputs a binary relevance vector \(\hat{y}_i(q) \in \{0,1\}^{K_i}\) per image. Calibrated scores from token logits use a recall-oriented hybrid rule: \(\hat{S}_i(q) = \{(i,k) \mid p_{i,k}(q) \ge \tau\} \cup \text{TopK}(\{p_{i,k}\}, K)\).

  2. Transcribe: Selected indices map deterministically to verbatim stored text: \(E = \bigoplus_{(i,k)\in\hat{S}(q)} s_{i,k}\). No free-form generation occurs.

Multi-resolution aging: Recent 5 interaction steps are stored at \(1024\times1024\) (\(n(r)=256\) visual tokens); all older history is downsampled to \(512\times512\) (\(n(r)=64\) tokens). Upon a retrieval hit on a low-resolution item (\(\ell_i > \ell_{\min}\)), the image is re-rendered on demand at full fidelity (active recall upscaling).

Training: The vision encoder \(\theta_{\text{vis}}\) is frozen; the decoder \(\theta_{\text{dec}}\) is fine-tuned via LoRA on HotpotQA (distractor subset, 3 epochs) with weighted binary cross-entropy \(\mathcal{L}_{\text{BCE}}\) using \(w_+ = 2.0, w_- = 1.0\) to bias toward recall. A resolution curriculum randomly samples the tier \(\ell \sim \text{Categorical}(\pi)\) with \(\pi=[0.3, 0.7]\) over \(\{1024, 512\}\) to simulate aging during training.

Key Contributions

  • First memory framework that stores agent episodic history in the image modality, enabling lossless retention under fixed context budgets.
  • Locate-and-Transcribe paradigm: reduces hallucination by replacing free-form textual retrieval with binary index classification + deterministic text fetching.
  • Age-aware adaptive resolution with active recall upscaling, mimicking human vivid-to-fuzzy memory decay while remaining reversible on retrieval hits.
  • Weighted BCE training on repurposed HotpotQA with resolution curriculum to handle multi-resolution inference conditions.

Results

  • Mind2Web (Cross-Task, 4096-token budget):
  • Element Accuracy: 53.8% vs. ACON 49.1%, AWM 48.2%, Retrieval 41.3%, Zero-Shot 40.1%
  • Step Success Rate: 46.1% vs. ACON 41.4%, AWM 42.6%
  • Task Success Rate: 4.8% vs. ACON 4.1%, AWM 4.3%
  • AppWorld Average SR: 58.1% vs. ACON 56.2%, AWM 55.0%, Zero-Shot 41.9%
  • Hard SR: 30.8% vs. AWM 27.2%, Retrieval 21.4%
  • SoM ablation (Mind2Web): removing SoM and using text generation drops Ele Acc to 46.5%, Step SR to 39.2%, and triples inference latency to 5.3 s vs. 1.7 s for full model.
  • Static Low-Res baseline drops Step SR by 6.4 percentage points relative to the dynamic multi-resolution strategy.

Limitations

  • Depends on DeepSeek-OCR (3B) as backbone; generalizability to other VLMs is untested.
  • Fine-tuning on HotpotQA is a domain proxy; the gap between multi-hop QA retrieval and agent-trajectory retrieval is not fully analyzed.
  • Task Success Rate on Mind2Web remains very low (4.8%), indicating the memory bottleneck is partially resolved but task-level success still lags.
  • The aging policy is a hard two-level step function (recent 5 steps vs. all prior), not a continuous or learned decay.
  • Rendering and on-demand re-rendering introduce additional wall-clock latency not fully quantified beyond the per-step retrieval latency.
  • Evaluation limited to two benchmarks; broader coverage across domains is absent.

Relevance to Agentic AI / LLM Agents

OCR-Memory directly targets the long-horizon memory bottleneck that limits autonomous LLM agents — the fundamental tension between rich episodic history and finite context windows. By recasting memory retrieval as a visual grounding task rather than a text-generation task, it offers a principled, hallucination-resistant path to scaling agent memory without lossy summarization. The locate-and-transcribe paradigm is a general architectural pattern applicable to any agent architecture that can be augmented with a VLM retriever. The work also highlights that multimodal representations can serve as a compression medium for text-heavy agent logs, complementing broader trends around external memory systems (MemoryBank, Reflexion, AWM) and context-compression methods in the agentic AI literature.