Skip to content

VISTA: A Vision and Intent-Aware Social Attention Framework for Multi-Agent Trajectory Prediction

🕒 Published (v1): 2025-11-13 11:17 UTC · Source: Arxiv · Venue: WACV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VISTA is a recursive, goal-conditioned transformer for joint multi-agent trajectory prediction that decouples destination-goal estimation from local trajectory generation, then fuses both via cross-attention. It adds a social-token attention module enabling interpretable pairwise influence maps between agents. On MADRAS and SDD benchmarks it reaches near-zero collision rates while matching or surpassing state-of-the-art ADE/FDE.

Problem

Existing trajectory prediction models either model agent interactions well but lack long-term goal integration, or focus on individual goals without modeling dynamic social interactions — yielding socially unrealistic (collision-prone) predictions. Interpretability is also limited, hindering deployment in safety-critical systems.

Method

VISTA consists of two sequentially connected, jointly trained modules:

  1. Goal Prediction Module (GPM): A U-Net decoder ingests past trajectories and a pretrained semantic segmentation map to produce per-agent spatial goal heatmaps. A differentiable softargmax converts heatmaps to continuous 2D goal coordinates. At test time, Test-Time Sampling Trick (TTST) + K-means diversifies candidate goals.

  2. Trajectory Prediction Module (TPM): A recursive transformer decoder runs from \(t = T_\text{obs}+1\) to \(T_\text{pred}\). At each step: (a) past positions are embedded with Hybrid Positional Encoding (sinusoidal + learnable offsets); (b) a cross-attention block fuses the goal token \(e^i_g\) with the past-trajectory token sequence, keeping trajectory decoding continuously goal-aware; (c) goal-aware features from all \(N\) agents are stacked and passed through multi-head self-attention (social attention), producing pairwise attention matrices \(A^{(t)} \in \mathbb{R}^{N \times N}\); (d) an MLP decodes per-agent displacement \(\Delta\hat{y}^i_t\), and positions are updated autoregressively. Weights are shared across time steps. Training uses a joint BCE loss on goal heatmaps plus MSE on trajectory positions.

Key Contributions

  • End-to-end framework unifying goal-heatmap conditioning, recursive multi-agent trajectory refinement, and learnable social tokens within a single transformer architecture.
  • Cross-attention goal–trajectory fusion at every recursive step (not a static prior), grounding each prediction in long-term intent throughout decoding.
  • Social-token attention producing stepwise pairwise attention matrices that are directly interpretable as agent-to-agent influence patterns.
  • Introduction of collision rate as a first-class evaluation metric alongside ADE/FDE/AUC, with a dataset-adaptive threshold calibrated to ground-truth proximity.

Results

  • SDD (standard): VISTA ADE/FDE = 28.4/57.4 px (vs. MART 37.6/78.5, TUTR 37.9/79.5); minADE/minFDE = 7.85/11.78 (best minFDE among all compared models); AUC = 244 (best); collision rate = 0% (MART: 1.5%, Y-Net: 8.11%).
  • MADRAS (high-density): ADE/FDE = 0.64/1.13 m (best); minADE/minFDE = 0.18/0.25 m (competitive with MART 0.17/0.24); AUC = 5.59 (best); collision rate = 0.03% (MART: 2.14%, Y-Net: 5.36%).
  • SDD long-term (5+30): best minFDE (65.13 px), KDE-NLL (9.09), and MissRate (0.002); collision rate 0.76% vs. Y-Net 1.45%.
  • Ablation: removing goal conditioning degrades minADE/minFDE from 7.85/11.78 to 25.79/49.39; removing social attention raises collision rate from 0% to 2.10%; hybrid PE over fixed-only reduces collision rate from 0.75% to 0%.

Limitations

  • Social attention has O(N²) complexity, limiting scalability to very large crowds; authors explicitly defer efficiency improvements to future work.
  • Evaluated only on pedestrian datasets (SDD, MADRAS); generalization to heterogeneous-agent scenes (vehicles + cyclists) is untested.
  • Goal heatmap quality is bounded by the pretrained segmentation backbone; no ablation on segmentation quality degradation.
  • Long-term prediction (5+30 horizon) shows slight minADE regression vs. Di-Long despite leading on other metrics, suggesting room for improvement at very long horizons.

Relevance to Agentic AI / LLM Agents

While VISTA operates in the physical-world trajectory domain rather than language-model agent pipelines, its architectural patterns are directly relevant to multi-agent coordination research: decoupling intent (goal) estimation from action (trajectory) execution mirrors the planner–executor separation common in LLM agent architectures. The social-token attention mechanism, which enables agents to attend to each other's representations and produce interpretable pairwise influence matrices, is an instance of agent-to-agent communication with built-in explainability — a property increasingly sought in cooperative LLM agent systems. The collision-rate metric also foreshadows the kind of safety-compliance evaluation needed when multiple autonomous agents share a workspace. Researchers building multi-agent LLM systems for embodied or robotics applications will find VISTA's goal-conditioned, interaction-aware design directly transferable.