Skip to content

ARCTraj: A Dataset and Benchmark of Human Reasoning Trajectories for Abstract Problem Solving

🕒 Published (v1): 2025-11-14 08:52 UTC · Source: Arxiv · Venue: KDD 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARCTraj is a dataset of ~10,000 human reasoning trajectories collected while solving ARC-AGI-1 tasks, structured as object-level, temporally ordered action sequences compatible with MDP formulation. Unlike prior ARC datasets that record only pixel-level edits or final outputs, ARCTraj captures intermediate reasoning steps at the semantic level (Move, Rotate, Flip, Copy), enabling trajectory-supervised learning for reinforcement learning, generative, and sequential modeling paradigms.

Problem

Existing ARC research relies on static input-output supervision, giving models no signal about how humans reason through abstract visual transformations. Prior human trajectory datasets (H-ARC, ARC-Interactive-History-Dataset) record low-level pixel edits with inconsistent formatting, making it difficult to extract structured reasoning or integrate with modern learning frameworks. This prevents models from learning the intermediate hypothesis-testing, attention allocation, and strategy formation that underlie human generalization.

Method

Trajectories were collected via the O2ARC web interface, where human solvers manipulate ARC grids using object-centric operations. Each action is represented as a triplet ⟨category, object, operation⟩ tied to a grid state and timestamp, forming MDP-compatible state-action pairs ⟨s_t, a_t, s_{t+1}⟩ with binary reward r_t ∈ {0,1}. Objects are perceptually meaningful contiguous regions confirmed by user selection rather than heuristic clustering. The dataset is accompanied by a unified reasoning pipeline: data collection → action abstraction → MDP formulation → downstream learning, with preprocessing paths defined for RL environments (ARCLE: grid+object+operation) and sequence models (Decision Transformer: grid+operation only). Three auxiliary knowledge signals are extracted from trajectories: (1) selection biases (spatial attention heatmaps showing compact 1×1–3×3 square/bar selection preferences), (2) color origins (66.5% of tasks resolved from test-input colors alone; 33.5% require example-output colors), and (3) shared intentions ((selection, operation) pairs that recur cross-participant, identifying strategy convergence).

Key Contributions

  • ~10,000 object-level, temporally ordered human trajectories across all 400 ARC-AGI-1 training tasks from 300+ participants (100 in the aligned 400-task split)
  • MDP formalization enabling direct integration with PPO, World Models, Decision Transformers, Diffusion agents, and GFlowNets
  • Three extractable auxiliary knowledge types (selection biases, color origins, shared intentions) quantified and analyzed
  • Empirical insight that 63.7% of object operations are preceded by a single selection, and >90% by ≤4 selections, revealing rapid exploratory attention patterns
  • Cross-trajectory grid convergence ratio of 43.7% (vs. 11.4% in H-ARC), indicating shared intermediate reasoning states
  • Public release: dataset, interactive viewer, and data collection platform

Results

  • Decision Transformers trained with ARCTraj intention cues gain 6–8 points over demonstration-only baselines
  • GFlowNet models augmented with ARCTraj selection priors produce trajectories with 14% higher diversity in valid transformations
  • ARCTraj contains 10,672 trajectories vs. H-ARC's 7,916, despite fewer participants (100 vs. 783), indicating deeper per-task coverage
  • Object-related action ratio: 37.7% in ARCTraj (excluding selection steps) vs. 0.9% in H-ARC
  • Representative solver performance ranges (from Table 5): PPO 55–70%, Decision Transformer 59–90%, Diffusion 77–92%, GFlowNet 10–100% (on ARC-related tasks; not all directly comparable)

Limitations

  • Only covers the 400 ARC-AGI-1 training tasks; no evaluation-split trajectories
  • Participant pool is moderate (300+ total, ~100 on aligned split); diversity of reasoning strategies may not fully generalize
  • Color origin information is inferred from observed patterns rather than explicitly logged at action time
  • Reported solver performance improvements (6–8 pt Decision Transformer, 14% GFlowNet diversity) are described without ablation details or statistical significance in the text
  • Trajectory-based tooling and benchmarks are described as "in early stages," limiting systematic evaluation
  • No tasks require colors exclusive to example inputs, reflecting a deliberate ARC design constraint that may limit generality of color-reasoning findings

Relevance to Agentic AI / LLM Agents

ARCTraj directly addresses a core challenge in agentic reasoning: learning how to reason through multi-step abstract problems rather than imitating final outputs. The MDP formulation with object-level actions maps naturally onto how LLM agents decompose and execute plans — the trajectory structure (attention → hypothesis → transformation → verify) mirrors the observe-reason-act loop in tool-using agents. The auxiliary knowledge signals (selection biases, shared intentions) offer a concrete recipe for injecting human-derived reasoning priors into agents tackling structured visual or symbolic tasks, which is directly relevant to work on LLM-based ARC solvers and test-time compute scaling. The dataset also serves as a benchmark for evaluating whether agents exhibit human-like intermediate reasoning states, not just correct final answers.