PM-Bench: Evaluating Prospective Memory in LLM Agents

🕒 Published (v1): 2026-07-14 05:57 UTC · Source: Arxiv · Venue: COLM 2026 · link

Why this paper was selected

COLM 2026; Saadia Gabriel; first rigorous benchmark for prospective memory in long-horizon agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PM-Bench is a text-based benchmark for evaluating prospective memory in LLM agents — the capacity to execute a deferred intention at the correct future cue or time while handling ongoing activity. Adapted from the Virtual Week cognitive-science paradigm, it exposes a precision–recall tension across all tested scaffolds, with the best configuration (GPT-5.4 + optional heartbeat) reaching only 65.1% macro Set-F1.

Problem

Existing LLM agent benchmarks stress retrospective retrieval — whether past information can be recovered — but none cleanly isolate whether a stored intention is executed at the right future moment under ongoing distraction. Agent systems increasingly fail not from lack of knowledge but from failure to act on that knowledge at the correct cue, a gap that prior memory benchmarks (MemoryOS, MemGPT, etc.) do not diagnose.

Method

PM-Bench adapts the Virtual Week board-game paradigm into a fully automated text-based simulation. A single generated week spans 7 days and 80 steps (10–13 steps/day). At each step \(t\), the agent receives a narrative scene plus three mandatory ongoing-activity choices (A/B/C) and a menu of opaque prospective action handles mixing valid tasks with lures. Before committing, the agent may issue monitoring queries to 11 hidden state channels (email, clock, portals, etc.). The agent then outputs \(a_t = (c_t, A_t)\) — an ongoing-activity selection and a subset of prospective actions.

The due set \(D_t = \{\tau \in X_t : \tau\text{ remains valid and its execution condition is satisfied at }t\}\) is ground-truthed by a deterministic replay evaluator. Performance is measured by set-level F1:

\[\text{Set-F1} = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}}\]

where TP/FP/FN are accumulated across all steps. Eight agent configurations are evaluated: single-agent baseline, todo-ledger single agent, three heartbeat variants (optional/auto-60m/auto-30m), hierarchical union-query multi-agent, and two replay ablations (majority vote, unanimous vote) over the multi-agent traces. Eight frontier model backbones are tested: GPT-5.4, GPT-5.3-Codex, Llama-3.3-70B, Mistral-Large-3-675B, Mistral-Small-3.2-24B, Qwen3-8B/14B/32B.

Key Contributions

• PM-Bench benchmark: 81 scored tasks (57 event-based, 26 time-based), 74 lures, 11 state channels, cross-day obligations, and 11 dynamic update events (2 cancel, 3 override, 6 reschedule) in a solvability-validated synthetic week.
• Evaluation protocol: set-level F1 as primary metric; auxiliary fine-grained diagnostics including miss rate, late completion, update/dependency violations, monitoring overhead (query count, false positives).
• Empirical comparison: 64 runs (8 models × 8 setups) revealing that no single scaffold dominates across models or sub-task types, and that proactive hidden-channel monitoring is the hardest bottleneck.

Results

• Best overall: optional-heartbeat single agent at 65.1% macro Set-F1 (GPT-5.4 backbone); todo-ledger second at 62.8% with fewest false positives (134 FP).
• Per-model peaks: GPT-5.3-Codex + single baseline 78.9%; GPT-5.4 + optional heartbeat 79.1%; Qwen3-8B + optional heartbeat 71.9%.
• Hierarchical multi-agent: 45.2% macro Set-F1 despite issuing 1661 total queries — worst F1 among primary configurations, exposing that query volume does not translate to execution quality.
• Monitoring hit rates (optional heartbeat): 80.1% on narrative-cued (no monitoring required), 54.7% on clock-monitoring-required, 10.0% on hidden-channel-monitoring-required.
• Auto-heartbeat 30m: highest hidden-channel hit (15.8%) and update hit (47.2%) but 489 FP, yielding only 57.8% macro Set-F1.
• Majority-vote replay: 37.2% macro Set-F1 with 655 FP — shows action aggregation, not query acquisition, is the binding bottleneck.
• No single scaffold is universally best across backbone models (Table 3 shows different backbones prefer different interventions).

Limitations

• Only one generated week (single scenario) is released; generalization across diverse scenario seeds is unvalidated.
• All tasks are synthetic: real-world prospective memory complexity (ambiguous cues, conflicting priorities, embodied context) is not captured.
• Evaluation is text-only and tool-free; results may not transfer to richer agentic environments with external APIs or multi-modal inputs.
• The benchmark is currently not multilingual and reflects a specific daily-life activity framing.
• Hidden-channel monitoring results are extremely low (≤15.8% hit rate) but the paper does not fully analyze why; qualitative breakdown is deferred to the appendix.
• Lure design (74 distractors) is fixed and deterministic; adversarially adaptive lures are not explored.

Relevance to Agentic AI / LLM Agents

Prospective memory — deferring action to the correct future cue while staying on-task — is a foundational requirement for any agent that must manage long-horizon schedules, reminders, or condition-triggered workflows, all common in real assistant deployments. PM-Bench isolates this capability cleanly from retrospective retrieval, exposing a critical failure mode missed by prior agent benchmarks: LLMs can store an intention but systematically miscalibrate when to execute it, producing either over-aggressive false alarms or conservative misses. The precision–recall tension documented here — where monitoring-heavy scaffolds raise recall at severe precision cost — is a concrete architectural design constraint for multi-step agent systems. The finding that no single inference-time scaffold generalizes across backbone models suggests that prospective behavior requires backbone-level training interventions, not just prompting.