A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments¶

🕒 Published (v1): 2025-12-15 16:43 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes the first mechanistic neural model of human mental rotation—the ability to compare 3D objects from different viewpoints—composed of an SO(3)-equivariant spatial encoder, a neuro-symbolic object descriptor, and an MLP decision agent that iteratively aligns objects via discrete quadrant-based rotation actions in latent space. The model design is informed by novel VR experiments where subjects manipulated Shepard-Metzler shapes with a joystick, revealing that humans use ballistic, discrete rotational actions guided by coarse symbolic object descriptions. The model achieves 96.13% accuracy on the task and reproduces key behavioral signatures including action counts and partial reaction-time linearity.

Problem¶

Despite decades of psychophysics on mental rotation, no prior model offered a mechanistic neural account of how the brain performs it—only behavioral curve-fitting. Modern computer vision solves 3D object matching but cannot explain why reaction time scales linearly with angular disparity, why subjects use surprisingly few, coarse rotation actions, or why in-depth and in-plane rotations cost the same time. The gap is a generative, process-level model constrained by both behavioral and neural evidence.

Method¶

Three independently trained modules, chained at inference as an agentic loop:

Equivariant Neural Renderer (EqNR): A 2D→3D convolutional autoencoder trained on pairs of 2D views of the same object at different poses. The latent space is SO(3)-equivariant (applying a rotation matrix to latent spatial indices produces the rotated view), achieved without any 3D supervision. Trained on 50,000 image pairs of Shepard-Metzler objects.
Vision Symbolic Model (VSM): A ViT encoder + autoregressive transformer decoder that maps EqNR's 3D latent representations into sequential symbolic object descriptions. The symbolic code encodes the 9 cube-to-cube transitions (U/D/B/F/L/R) starting from the nearest cube. Under the "Quadrant Hypothesis," the symbolic description is constant within each 90° azimuthal quadrant, so only 4 distinct codes exist per object. Trained on 201,600 image–description pairs.
MLP Decision Agent: A 3-hidden-layer MLP operating on concatenated Siamese symbolic logit vectors from both views. Outputs one of five classes: SAME, MIRROR, 1-quadrant CW, 1-quadrant CCW, or 2-quadrants. At inference, when a rotation action is predicted, it is applied to the EqNR spatial representation of the source view (not the target), which is re-encoded through VSM and passed back to the MLP—an iterative agentic loop that terminates on a similarity decision or after 6 actions. Trained on 38,400 tasks.

VR experiments (n=19, 4 excluded) provided an Action setup (joystick rotation, object hidden during rotation) and No-Action setup, measuring both reaction times and explicit action sequences to constrain model design.

Key Contributions¶

First mechanistic neural model of human mental rotation validated against behavioral data.
Novel interactive VR paradigm that exposes subjects' discrete rotation strategies directly (action timing, count, and final angular residual).
Quadrant Hypothesis: formalizes that Shepard-Metzler symbolic descriptions are invariant within 90° azimuthal quadrants, explaining why coarse, ~73° ballistic actions suffice.
Three-module neuro-symbolic agentic architecture combining equivariant spatial representation, autoregressive symbolic encoding, and iterative decision-action loop.
Systematic ablation study demonstrating each component is necessary for both accuracy and behavioral plausibility.

Results¶

Overall accuracy: 96.13% (96.39% match, 95.87% mismatch) vs. humans 91.14% (No-Action) / 95.33% (Action).
Action count match: model takes 0.00 actions at 0°, 0.71 at 60°, 1.03 at 120° and 180°; humans average 1.05 actions total (Action setup), dominated by 0-action trials at 0° and 1-action trials at other disparities.
Reaction time: model partially explains linear RT scaling from 0°→60°→120° through action count, but cannot distinguish 120° from 180° (both require ~1 action).
Ablations:
Siamese feed-forward baselines (ResNet/ViT on raw images): near-perfect on training objects, fails on test objects for in-depth rotations.
Without EqNR (direct image input to VSM): symbolic encoder fails, task unsolvable.
Without VSM (frozen random 3D conv or patch embedding only): ~38–36% accuracy; with ViT encoder retained but without autoregressive decoder: ~90.71% accuracy with reduced action efficiency.
Without actions (direct similarity prediction from symbolic concatenation): 97.03% accuracy but zero actions taken—cannot account for human behavior.
1-layer MLP with actions: 96.05% accuracy, maintains correct action counts.

Limitations¶

Model is deterministic; does not capture trial-by-trial variability in human action counts or reaction times.
Cannot explain the reaction-time difference between 120° and 180° disparities.
No neural mechanism proposed for the SO(3) rotation operation on latent indices (left to speculation in appendix).
Symbolic coding scheme (cube-transition sequences) is a tractable proxy and may not reflect actual human internal representations.
No saccadic eye-movement mechanism; autoregressive token sequence is a coarse analogue of eye-tracking data.
Evaluated exclusively on Shepard-Metzler shapes; generalization to arbitrary 3D objects untested.
Small VR sample (15 usable subjects); no significant correlation found between human RT and action count at fixed disparity; error consistency between model and humans is weak.

Relevance to Agentic AI / LLM Agents¶

The third module is explicitly framed as an agentic system that selects and executes rotation actions in a 3D latent space iteratively until a decision criterion is met—a micro-scale instance of planning via latent simulation, directly relevant to world-model-based agents. The architecture demonstrates that decomposing perception into equivariant spatial representations and symbolic object descriptions enables a lightweight MLP to serve as a decision/action agent, a neuro-symbolic pattern increasingly explored in embodied AI and tool-using LLM agents. The quadrant-based action discretization—reducing a continuous rotation space to four symbolic states and three action classes—is a concrete example of how structured world models can collapse large action spaces into tractable, efficient sequential decision-making. For researchers tracking agentic AI, this paper provides empirical and architectural evidence that hybrid spatial-symbolic representations are necessary (not merely sufficient) for an agent to exhibit human-like sequential reasoning, even on a tightly scoped visuospatial task.