Coherent Multi-Agent Trajectory Forecasting in Team Sports with CausalTraj¶

🕒 Published (v1): 2025-11-23 02:24 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CausalTraj is a temporally causal, likelihood-based model for joint multi-agent trajectory forecasting in team sports that explicitly models the joint distribution over all agents' futures rather than marginal per-agent distributions. It achieves state-of-the-art performance on joint accuracy metrics (minJADE, minJFDE) across NBA, basketball, and football datasets while remaining competitive on standard per-agent metrics.

Problem¶

Existing trajectory forecasting models are designed and evaluated primarily on per-agent metrics (minADE, minFDE), which select each agent's best prediction independently across k samples—allowing trajectories from incompatible scenarios to be mixed. This creates a structural incentive to model marginal distributions rather than the true joint distribution, producing individually plausible but collectively incoherent multi-agent futures. Joint metrics (minJADE, minJFDE) that evaluate the best full scenario sample have been underutilized in sports analytics.

Method¶

CausalTraj factorizes the joint future distribution autoregressively over timesteps: p(X_{P+1:T} | X_{1:P}) = ∏ p(X_{t+1} | X_{1:t}), predicting per-step displacement deltas for all N agents simultaneously. At each step, the model outputs a mixture of M=8 Gaussians over collective displacements, with block-diagonal per-agent covariances parameterized via Cholesky factors and shared mixture weights that couple agent outcomes.

Architecture components: - Agent history encoder: either a causally adapted PointNet (lookback max-pooling) or Mamba2 layers applied independently per agent - Inter-agent relation encoder: standard transformer self-attention across agents at each timestep, augmented by a Spatial Relation Transformer Encoder (SRTE) that constructs pairwise mesh tensors M[q,k] = concat(x_q − x_k, z_q, z_k) to expose exact relative displacements - Scene aggregation & prediction head: agent features concatenated into a scene-level representation, passed through a 3-layer MLP to produce mixture parameters - Training uses NLL loss with entropy regularization (λ=0.05) to prevent component collapse; teacher forcing enables parallel training

Key Contributions¶

Causal (autoregressive) multi-agent trajectory model that directly optimizes the joint distribution without a compressed global latent
Spatial Relation Transformer Encoder augmenting attention with explicit pairwise positional offsets
Causally adapted PointNet encoder preserving temporal ordering via lookback max-pooling
Emphasis on joint metrics (minJADE, minJFDE) as primary evaluation criteria for sports trajectory forecasting
State-of-the-art joint-metric results on NBA SportVU, Basketball-U, and Football-U benchmarks

Results¶

NBA SportVU (k=20, Table 1): - minJADE20 at 4.0s: CausalTraj (C-PointNet) 1.34 m, Mamba2 1.38 m vs. MoFlow 1.69 m, LED 1.63 m, GroupNet 2.12 m - minJFDE20 at 4.0s: C-PointNet 2.47 m, Mamba2 2.57 m vs. MoFlow 3.31 m - Per-agent: MoFlow leads at longer horizons (minADE20 at 4.0s: 0.71 m vs. CausalTraj 0.77 m); CausalTraj slightly better at ≤2.0s

Basketball-U (20-frame horizon, Table 2): - minJADE20: Mamba2 0.50 m vs. MoFlow (joint obj.) 0.63 m, Sports-Traj 1.52 m - minJFDE20: Mamba2 0.99 m vs. MoFlow (joint obj.) 1.62 m - Per-agent: comparable to MoFlow default (0.56/0.71 vs. 0.50/0.61)

Football-U (20-frame horizon, Table 2): - minJADE20: Mamba2 1.12 yd vs. MoFlow (joint obj.) 1.19 yd, Sports-Traj 3.66 yd - minJFDE20: Mamba2 2.68 yd vs. MoFlow (joint obj.) 2.87 yd

Ablation (Basketball-U, joint metrics, 20-frame): - Full model (Mamba2): minJADE20=0.97, minJFDE20=1.77 - No SRTE: 0.99/1.81 - Single Gaussian: 1.03/1.86 - Component-mean sampling only: 1.05/2.13

Limitations¶

Occasional physically implausible behaviors remain (player-ball spatial gaps, ball boundary violations), attributed to limited cross-agent covariance modeling capacity (within-mixture-component covariance is block-diagonal, not full cross-agent)
Per-agent accuracy lags behind MoFlow at longer prediction horizons (3–4 s), as CausalTraj does not explicitly optimize for per-agent sample diversity
Model is only ~3M parameters; scalability and behavior under larger capacity untested
Joint metrics still measure accuracy against ground truth, not perceptual coherence directly; the gap between quantitative joint metrics and true realism remains an open problem
Evaluation limited to 2D coordinate data; no ball/player semantic state modeling

Relevance to Agentic AI / LLM Agents¶

CausalTraj directly instantiates the autoregressive, causally-structured generative paradigm—the same architectural principle underlying LLMs—applied to multi-agent behavior modeling. The core insight that per-agent marginal optimization fails to capture joint coordination is structurally analogous to challenges in multi-agent LLM systems where independently optimized agents produce individually coherent but collectively inconsistent plans. The model's approach of learning a shared joint distribution with coupled mixture weights, rather than composing independent marginal predictions, offers a concrete design principle for multi-agent coordination under uncertainty. For researchers building LLM-based agents that must coordinate across sub-agents or reason about joint futures (e.g., robot teams, multi-agent planning), CausalTraj provides an empirically grounded demonstration that autoregressive joint modeling outperforms marginal-then-compose strategies on coherence-sensitive tasks.