Skip to content

PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning

🕒 Published (v1): 2026-07-22 12:11 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PRO-LONG is a minimal agent harness that replaces heuristic memory systems with a single design choice: append every observation, action, and outcome to a plain log file, then let a coding agent search that log programmatically. On ARC-AGI-3, it matches or beats specialized harnesses (up to 76.1% pass@1) while consuming 4.2–5.8× fewer tokens, and reaches 97.4% best@2 with Fable 5 at 2,000 actions for $1,750.

Problem

Long-horizon LLM agents suffer from a fidelity–tractability tradeoff in context management: systems that compress or summarize history (scratchpads, embeddings, tiered memory) lose details whose relevance is only apparent in hindsight, while keeping everything in the active context window degrades with scale ("context rot"). Existing specialized harnesses are also typically tuned to a single model and require complex custom tooling (e.g., 12-tool MCP servers, multi-subagent orchestrators), making them brittle and expensive.

Method

PRO-LONG wraps a standard coding agent (Claude Code or Codex) with two operations:

  • Write: The harness appends a structured entry to logs.txt after every action — header (action number, level, attempt, score), the agent's summarized plan, the chosen action, and the resulting board state. No compression, no filtering, no learned decisions.
  • Read: The agent uses its native programmatic tools (file read → grep/regex → Python) to search the log on demand. No vector database or custom retrieval index; ordinary shell utilities and Python are sufficient for logs exceeding 100k+ lines.

The framework adds ~30 lines of prompt and zero custom tools to the base coding agent. The agent's active context window holds ~100k–1M tokens; the accessible log can grow to 10M+ tokens and is queried lazily via code.

Key Contributions

  • Programmatic memory formalism: formalizes agent memory as an append-all write + code-based read, motivating simplicity, losslessness, and native compatibility with coding agents.
  • ARC-AGI-3 SOTA efficiency: 42.4% pass@1 with Opus 4.6 (beats Arcgentica by +3pp); 41.2% pass@1 / 60.1% best@5 with GPT-5.5 at 5.8× lower token cost than WorldModeler; 97.4% best@2 with Fable 5 at <1/3 the cost of Schema.
  • 18.0 pp average gain over the same base coding agent without the log, across frontier models.
  • Ablation evidence for design principles: programmatic tool depth (read → grep → Python) accounts for nearly all gains; workspace write/edit tools and persistent notes add negligible benefit; fuller logs widen best@k gaps, showing they expand the solvable game set rather than just improving already-solvable games.

Results

  • PRO-LONG vs. no-log baseline: +15.7–21.0 pp pass@1 across GPT-5.5, Opus 4.6, Fable 5 (500-action budget).
  • GPT-5.5 / Codex: 41.2% pass@1, 60.1% best@5 vs. WorldModeler 45.1% pass@1 — using 5.8× fewer tokens; at best@5, PRO-LONG is 1.2× cheaper.
  • Opus 4.6 / Claude Code: 42.4% pass@1 vs. Arcgentica 39.0% pass@1.
  • Fable 5 / Claude Code, 500 actions: 76.1% pass@1, 82.1% best@2 vs. Schema 84.4% best@2 — using 4.2× fewer tokens (~400M token saving across the game set).
  • Fable 5, 2,000 actions: 94.6% pass@1 (\(1,500), 97.4% best@2 (\)1,750) vs. Schema's partial retained-run cost of $6,447 for 99.0% best@2.
  • Tool ladder ablation (GPT-5.5): read-only 23.1% → +grep 27.2% → +Python 38.3% → +write/edit 41.2%.
  • Workspace persistence ablation: clearing workspace every call drops no-log by ~4 pp but drops PRO-LONG by only 0.5 pp (41.2% → 40.7%).
  • m0r0 game: PRO-LONG 100% on all 5 runs; no-log 34.2%.

Limitations

  • Evaluated only on ARC-AGI-3 (25 public games); generalizability to other long-horizon benchmarks (e.g., NetHack) is asserted but not demonstrated with numbers.
  • Log size grows unboundedly; the paper notes tractability up to 100k+ lines but does not characterize failure modes at extreme scales or very long action budgets beyond 2,000 actions.
  • Fable 5 best@2 at 2,000 actions is a lower bound (some games run only once), so the 97.4% figure is not fully reproducible as stated.
  • PRO-LONG does not use vision; all baselines except WorldModeler also forgo vision, but vision may matter for some games.
  • Token costs for Schema comparison use only Schema's retained (best-of-two) run logs, making the cost comparison partially favorable to PRO-LONG by construction.
  • Run-to-run variance is high (GPT-5.5 pass@1 40.7 ± 3.6); confidence intervals are computed from only 2–5 replicates per model.

Relevance to Harnesses / Meta-Harnesses

PRO-LONG is a direct contribution to the harness design space: it demonstrates that a minimal, model-agnostic harness — one that adds only a structured log and a ~30-line prompt — can match heavily engineered, model-specific harnesses (multi-subagent orchestrators, 12-tool MCP servers, custom world-model scaffolds) at a fraction of the token cost. The key insight for harness designers is that the read mechanism (programmatic vs. retrieval-based vs. summarization-based) is the decisive design dimension, not the richness of custom tooling. PRO-LONG also surfaces a measurable benchmark for comparing harness strategies: the fidelity–tractability tradeoff can be directly quantified via the tool-ladder ablation (Table 1) and the workspace-persistence ablation (Table 2). For researchers building meta-harnesses that compose or select among sub-harnesses, PRO-LONG's lossless log is a natural substrate — it preserves the full trajectory without committing to any extraction policy, leaving downstream stages free to query arbitrarily.