Skip to content

LIME: Learning Intent-aware Camera Motion from Egocentric Video

🕒 Published (v1): 2026-07-02 16:48 UTC · Source: Arxiv · link

Why this paper was selected

Active vision for robotics; intent-aware camera motion from egocentric video, novel VLM application

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LIME formulates language-conditioned camera motion generation: given an RGB observation and a free-form intent, predict a relative \(SE(3)\) target camera pose that acquires intent-relevant visual evidence. It mines ~3M intent-conditioned training examples from passive egocentric video using a VLM labeler and trains a Qwen3-VL-4B-Instruct backbone coupled with a continuous flow-matching pose head. On a 425-example photorealistic indoor benchmark, LIME achieves 47.7% overall success rate versus 24.3% for the best baseline.

Problem

Active perception—moving the camera before acting—lacks a language-conditioned formulation as a first-class action. Vision-language navigation (VLN) maps language to base motion; vision-language-action (VLA) models map language to end-effector control; neither exposes "how should the camera move to resolve this intent?" as a standalone interface. The core difficulty is that the desired next viewpoint depends on latent perceptual intent (inspect, reveal occluder, explore), and the same intent can be satisfied by multiple geometrically distinct target poses, making this a multimodal conditional distribution over \(SE(3)\) rather than a deterministic regression.

Method

Data mining. Temporally ordered start–goal frame pairs \((I_s, I_g)\) are sampled from RoomTour3D and Nymeria. A structured VLM labeler receives \(I_s\), \(I_g\), and a compact pose summary of \(T_{gs}\) (translation direction, distance, rotation angle), and produces: motion type, newly visible objects, enhanced views, spatial anchors, an observation-gain description \(g\), and \(m\) plausible intent candidates \(\{x_i\}\). Each transition is unrolled into \(m\) training examples \((I_s, x_i, g, T_{gs})\), yielding ~3M examples total.

Model (LIME). Given \((I_s, x)\), a Qwen3-VL-4B-Instruct backbone produces hidden states \(H_x = \text{VLM}_\theta(I_s, x)\) and autoregressively decodes the observation-gain description: $\(p_\theta(g \mid I_s, x) = \prod_k p_\theta(g_k \mid g_{<k}, H_x).\)$ The gain-conditioned hidden sequence \(H_{x,g} = [H_x; H_{g|x}]\) conditions a flow-matching pose head. The pose is parameterized as \(y = \psi(T_{gs}) \in \mathbb{R}^9\) (3D translation + first two columns of the rotation matrix). Using \(x\)-prediction flow matching with \(z_t = (1-t)\epsilon + ty\), the head predicts \(\hat{y}_\phi = F_\phi(H_{x,g}, z_t, t)\) with loss \(\mathcal{L}_\text{pose} = \|\hat{y}_\phi - y\|_2^2\). The full training objective is \(\mathcal{L} = \mathcal{L}_\text{gain} + \lambda_\text{pose} \mathcal{L}_\text{pose}\). The vision encoder is frozen; the multimodal projector, LM, and flow head are trained. At inference, gain-token hidden states are cached to avoid a second VLM forward pass.

Benchmark. 425 examples on InteriorGS (1K-scene real-world indoor 3DGS) across three intent families: Target-approaching, Exploration, and Perspective-shift. Evaluation uses a budgeted multi-step protocol (6 m translation, 600° rotation budget); success is judged by Gemini-3.1-Pro-Preview against the intent (not pose matching).

Key Contributions

  • Formalization of intent-aware camera motion generation as predicting \(p_{\theta,\phi}(T_{gs} \mid I_s, x, g)\) over relative \(SE(3)\) poses jointly with an observation-gain language description.
  • LIME: a VLM backbone with an autoregressive language interface for observation-gain and a continuous flow-matching pose head, enabling multimodal pose hypothesis generation.
  • A scalable data-mining pipeline that converts passive egocentric video into ~3M intent-conditioned camera-motion examples without teleoperated demonstrations.
  • A photorealistic benchmark (425 examples, 3 intent families) on real-world 3DGS scenes with a unified budgeted multi-step evaluation protocol.
  • Real-robot validation on Boston Dynamics Spot and integration with a downstream manipulation policy (VidBot) as a reusable active-perception module.

Results

  • Overall SR: LIME 47.7% vs. VG-AVS 24.3% (+23.4 pp), JanusVLN 21.9%, Uni-NaVid 12.6%, VLMnav 4.1%.
  • Overall CA-SR: LIME 34.4% vs. VG-AVS 16.5%, JanusVLN 19.3%.
  • Target-approaching SR: LIME 45.8% vs. VG-AVS 8.6%, JanusVLN 2.0%.
  • Exploration SR: LIME 51.4% vs. JanusVLN 34.5%, VG-AVS 30.0%.
  • Perspective-shift SR: LIME 45.8% vs. VG-AVS 36.4%, Uni-NaVid 26.5%.
  • Gemini–human agreement: 91.3% overall (85.3–98.0% per intent family) on 90-example balanced audit subset (450 trajectory labels).

Limitations

  • Benchmark is small (425 examples) and confined to indoor scenes; outdoor or unstructured environments are not evaluated.
  • Real-robot experiments use a LoRA-adapted checkpoint, indicating the base model does not directly transfer without adaptation.
  • Success judgment relies on Gemini-3.1-Pro-Preview, introducing dependency on a proprietary third-party model.
  • Training data is limited to two egocentric datasets (RoomTour3D, Nymeria); generalization to domains with different motion statistics is not characterized.
  • The paper does not present ablations on the observation-gain auxiliary loss or flow-matching vs. regression baselines in the visible text.

Relevance to Vision-Language Models

LIME directly extends VLMs beyond passive perception into active spatial reasoning, using a pretrained VLM (Qwen3-VL-4B) as the backbone for grounded intent understanding and then coupling it to a geometric action head—a concrete instantiation of VLMs as embodied action generators. The observation-gain auxiliary language objective shows how language-space supervision can improve the hidden representations used by a continuous geometric head, a pattern applicable to other VLA architectures. The egocentric data-mining approach—using one VLM to generate training supervision for another—is directly relevant to scalable grounding of VLMs without human annotation. For researchers tracking VLMs, this paper demonstrates a pathway from passive visual understanding to intent-conditioned active control, bridging VLN, VLA, and active perception under a unified language interface.