Skip to content

Agentic Time Machine as an Infrastructure for Future-Event Forecasting

🕒 Published (v1): 2026-06-19 00:55 UTC · Source: Arxiv · link

Why this paper was selected

Time-machine infrastructure for forecasting agents; harness-as-temporal-scaffold framing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agentic Time Machine (TM) is an LLM-based evaluation infrastructure that reconstructs the historical web state at any past timestamp by filtering out post-cutoff and answer-revealing content from live web tool calls, enabling fast and leakage-free retrospective replay of future-event forecasting tasks. Built on top of TM, a planner-solver-aggregator multi-agent framework decomposes each forecasting question into parallel analytical angles and achieves state-of-the-art on FutureX-Past, Polymarket, and the live FutureX leaderboard (1st overall as of June 17, 2026).

Problem

Evaluating forecasting agents faces a fundamental efficiency–fidelity trade-off: live benchmarks (FutureX, ForecastBench) require waiting days/weeks for event resolution, while frozen-snapshot benchmarks sacrifice dynamic web access and suffer from post-hoc answer leakage through rolling web content (e.g., a stable Yahoo Finance URL whose OHLC snippet extends to today). Naive date-based URL filtering is insufficient because pages carry rolling content behind stable, old-indexed URLs, causing scores in the "frozen" regime to be artificially inflated.

Method

Agentic Time Machine (TM): wraps every web tool call (search and page-fetch) with an LLM-based leakage filter \(\phi(x \mid q, t, a^\star)\) that checks each returned snippet or page for two leakage signs: (1) a direct answer leak (content states or implies \(a^\star\)) and (2) a post-date info leak (content carries a time signal later than cutoff \(t\)). The agent's effective environment becomes:

\[\hat{E} = E_{\text{TM}} = \bigcup_{\text{calls}} X^{\text{TM}}, \quad X^{\text{TM}} = \{x \in X : \phi(x \mid q, t, a^\star) = \text{KEEP}\}\]

This approximates \(E_t\) (the web as it stood at cutoff \(t\)) while retaining open-ended, dynamic tool interaction.

Planner-Solver-Aggregator framework: - Planner: executes exploratory searches, then outputs a JSON plan with task understanding, causal factors, confirmed historical patterns, and 2–4 solver angle profiles (each specifying a distinct evidence path and search focus), plus aggregation advice. - Solvers: 2–4 agents run in parallel on separate threads, each following only its own angle spec; each returns a structured JSON evidence report. - Aggregator: receives all solver reports, resolves conflicts with targeted searches, and outputs a final answer. No fine-tuning or additional tools beyond the single-agent baseline.

Key Contributions

  • Agentic Time Machine infrastructure: LLM-filter-based web replay that blocks answer leakage while preserving dynamic web interaction—combining the fast feedback loop of frozen replay with the realism of live evaluation.
  • Planner-solver-aggregator multi-agent framework: structured decomposition into diverse analytical angles with parallel evidence gathering and reconciliation, requiring no extra training or tools.
  • Empirical validation: TM reduces scores by 25–37 points across five LLM backbones, confirming leakage suppression; offline TM scores correlate strongly with live FutureX leaderboard rankings.
  • SOTA results: highest weighted score on FutureX-Past (39.51 vs. SC@5 36.81) and Polymarket (64.99 vs. SC@5 62.26); #1 on FutureX eight-week overall leaderboard as of June 17, 2026.

Results

  • TM leakage validation: Without TM, five backbones score 60.64–69.11 on FutureX-Past; with TM, scores drop to 22.80–35.02 (Δ = −25 to −37 points), consistent across all model families.
  • FutureX-Past (offline, TM on): Ours 39.51 > SC@5 36.81 > SC@3 36.04 > best single-agent (DeepSeek-V4-Flash) 35.02.
  • Polymarket (offline, TM on): Ours 64.99 > SC@5 62.26 > SC@3 60.99 > best single-agent (GLM-5) 59.24.
  • Tool-call efficiency: Ours averages 47.1 tool-call rounds per question vs. SC@3 at 51.4 and SC@5 at 81.0, achieving higher Polymarket score with fewer calls.
  • Live FutureX leaderboard: Best average rank over four consecutive weekly rounds including 1st in May Week 1; 1st overall on eight-week cumulative leaderboard as of June 17, 2026.
  • Web tools vs. closed-book: Adding TM-wrapped tools improves every backbone by 13–25 weighted points on FutureX-Past; even the weakest tool-equipped agent (22.80) outperforms the strongest closed-book model (15.06).
  • Level breakdown: L4 (quantitative/open-ended) shows the largest TM-induced drop (scores fall to 12–29); L1 (few-option) shows smallest drop (scores remain 68–72 due to random-guess floor).

Limitations

  • TM approximates \(E_t\) but is not exact: pre-cutoff articles that hint at the answer may be over-blocked, and post-cutoff pages that happen to reveal no leakage signs may slip through.
  • A single date stamp per URL is insufficient for rolling-content pages; the LLM filter is a heuristic workaround, not a guaranteed solution.
  • TM requires knowledge of \(a^\star\) (the resolved answer) at evaluation time, so it cannot be applied to truly live, unresolved questions.
  • Dataset sizes are small (61 questions for FutureX-Past, 80 for Polymarket), limiting statistical power.
  • Performance on L4 (quantitative forecasting) remains very low even with tools (~14–31 points weighted), indicating fundamental difficulty with hard open-ended prediction under TM.

Relevance to Harnesses / Meta-Harnesses

Agentic Time Machine is directly a harness-infrastructure contribution: it wraps and intercepts all web tool calls in an agent evaluation loop, functioning as a programmable middleware layer that reshapes the information environment seen by any downstream agent—exactly the pattern of a meta-harness that sits above task-level agents and controls what they can perceive. The planner-solver-aggregator framework is itself a meta-harness for multi-agent orchestration, dynamically generating the pipeline topology (number and role of solvers) per question rather than fixing it at design time. Together they demonstrate a two-level harness stack—environmental simulation at the infrastructure level and dynamic workflow orchestration at the agent level—a pattern directly relevant to researchers building configurable evaluation or execution harnesses for agentic systems.