Skip to content

FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Live benchmark for future prediction testing LLM agent reasoning under uncertainty

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FutureX is a live, contamination-free benchmark for evaluating LLM agents on future prediction tasks, continuously collecting ~500 events/week from 195 curated websites across 11 domains. It evaluates 25 models spanning base LLMs, search-augmented LLMs, and open/closed-source Deep Research agents. The benchmark reveals a steep performance gap between current agents and human expert-level forecasting, especially on high-volatility open-ended tasks.

Problem

Existing LLM benchmarks assess static knowledge or controlled tool use, and existing forecasting benchmarks (ForecastBench, FutureBench) suffer from: (1) reliance on prediction-market events that are simple and dominated by binary yes/no questions; (2) temporal leakage when backtesting on historical data; (3) no live updates, evaluating only vanilla LLMs without search, and covering very few events/domains. No large-scale, live, agent-centric benchmark existed for real-world future prediction requiring dynamic information gathering, synthesis, and probabilistic reasoning.

Method

FutureX operates as a fully automated daily pipeline with four stages:

  1. Event database construction: An LLM-based agent (AIME) seeds 2,008 websites; LLM filtering (Seed1.5-Thinking + DeepSeek-R1) reduces these to 636; manual curation yields 195 high-quality sources spanning prediction markets, news, entertainment rankings, government data, and real-time financial platforms.
  2. Daily event curation: Each day, events are extracted from the 195 sources, subjected to event manipulation (randomization/perturbation) and filtering (removing easy, harmful, or subjective questions), producing ~500 new events/week across 11 domains.
  3. Agent daily prediction: On each event's start date, 25 models are run automatically and predictions are stored.
  4. Answer daily acquisition: After the resolution date, the pipeline crawls the source website, extracts ground truth via Seed1.5-Thinking, and scores predictions. Answer acquisition success rate exceeds 97%.

Events are classified into four progressively harder tiers: Basic (Level 1, single-choice ≤4 options), Wide Search (Level 2, multi-choice exhaustive), Deep Search (Level 3, open-ended low-volatility), and Super Agent (Level 4, open-ended high-volatility). Evaluation uses exact match and F1 for Levels 1–2; LLM-as-Judge and clipped MSE for Levels 3–4:

\[\text{Score}_{\text{L3/L4}} = \text{LLM-Judge}(prediction, ground\_truth)$$ $$\text{Score}_{\text{numerical}} = \max\!\left(0,\, 1 - \frac{(p - g)^2}{\sigma^2}\right)\]

Key Contributions

  • First large-scale, live, contamination-free benchmark for LLM agent future prediction, collecting ~500 events/week from 195 websites across 11 domains.
  • Four-tier difficulty taxonomy (Basic → Super Agent) explicitly mapping event type and volatility to required agent capabilities (planning, reasoning, search strength).
  • Comprehensive evaluation of 25 models across four agent classes: base LLMs, search-augmented LLMs, open-source Deep Research agents, and closed-source Deep Research agents.
  • Fully automated, closed-loop evaluation pipeline (daily event curation, agent execution, answer crawling, scoring) with no manual intervention after initial setup.
  • In-depth analyses of performance by difficulty, domain, model type, agent framework, and the effect of resolving events retrospectively (post-resolution date prediction).

Results

  • Evaluated on 1,272 events from July 20 – August 3 (across 25 models including Gemini-2.5-Pro, DeepSeek-V3, and multiple Deep Research agents).
  • Deep Research agents (both open- and closed-source) substantially outperform base LLMs and search-augmented LLMs, particularly on Level 3 and Level 4 tasks requiring multi-step planning and synthesis.
  • Performance degrades sharply from Level 1 → Level 4; Super Agent tier (high-volatility, open-ended) remains extremely difficult for all current models.
  • Search and planning capabilities have statistically significant positive impact on performance; model type (reasoning vs. base) is also a significant factor (Section G.2).
  • Agents given access to answers after the resolution date (retrospective setting, Appendix G.3) score markedly higher, confirming that the live setting is genuinely harder than post-hoc evaluation.
  • Specific accuracy numbers for individual models are truncated in the provided text.

Limitations

  • Evaluation delay of up to one week means near-real-time performance cannot be assessed instantaneously.
  • Ground-truth acquisition depends on website crawlability; anti-bot measures can cause failures (mitigated by excluding inaccessible sites, but introduces selection bias toward crawlable sources).
  • The 195-website pool, while broad, skews toward English-language and web-accessible domains; less coverage of private or paywalled data sources.
  • LLM-as-Judge scoring for open-ended levels introduces its own reliability concerns (judge model bias, inconsistency).
  • Binary yes/no events were deliberately downsampled, which may underrepresent a realistic distribution of real-world forecasting questions.
  • Human expert baseline performance is not reported in the provided text, making it difficult to quantify the gap precisely.

Relevance to Agentic AI / LLM Agents

FutureX directly operationalizes the core competency stack of advanced LLM agents—planning, tool-augmented search, multi-source synthesis, and probabilistic reasoning under uncertainty—in a genuinely live, non-contaminated setting, which static or backtested benchmarks cannot provide. The four-tier difficulty taxonomy gives researchers a structured ladder to diagnose where their agent architectures break down, from simple retrieval (Level 1) to expert-level uncertainty-aware forecasting (Level 4). The comprehensive comparison of Deep Research agent frameworks against base and search-augmented LLMs provides empirical grounding for understanding which architectural choices (tool integration, planning loops) drive real-world agentic capability. For the agentic AI community, FutureX sets a renewable, scalable evaluation target analogous to what SWE-bench did for coding agents—but for open-domain, high-stakes reasoning in dynamic environments.