Skip to content

Focus-Then-Reuse: Fast Adaptation in Visual Perturbation Environments

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Focus-Then-Reuse (FTR) addresses sim-to-real visual perturbation in reinforcement learning by training a segment selector — guided jointly by a VLM and environment rewards — to isolate task-relevant objects, then feeding only those objects to a frozen, simulation-trained policy. This hierarchical "focus then reuse" design avoids modifying the base policy while achieving rapid domain adaptation. FTR reaches SOTA performance on 10/11 DMC and Franka robotics tasks, converging within 50k adaptation steps.

Problem

Policies trained in clean simulation degrade severely when deployed in environments with distracting video backgrounds. Existing approaches either train directly in visually perturbed environments (costly, low generalization), rely on data augmentation (fails on unseen disturbances), or use adaptation methods (PAD) that cannot recover clean-environment performance in complex, near-real-world video backgrounds.

Method

FTR is a two-stage hierarchical framework:

Focus stage: At each timestep, a frozen SAM 2 segmentation model decomposes the disturbed observation into \(k\) binary-masked segments. A trainable segment selector \(\pi^h\) (Gaussian policy over segment indices) picks which segments are task-relevant, outputting a union mask \(o_t^{\text{sel}}\). Between selection calls (interval \(T_{\text{sel}}=20\) steps), a frozen SAM 2 tracker propagates the previously selected masks for temporal consistency, preventing unstable RL signals from frame-by-frame re-selection.

Reuse stage: The frozen, DrQ-v2-pretrained source policy \(\pi^l\) receives the last 3 frames of \(o_t^{\text{sel}}\) and outputs actions. \(\pi^l\) is never updated.

Segment selector training: \(\pi^h\) is trained with a combined loss \(L = \eta_{\text{SL}} L_{\text{SL}} + \eta_{\text{RL}} L_{\text{RL}}\). \(L_{\text{SL}}\) uses Qwen-VL-Max labels (which segments are task-relevant) with a margin-regularized loss instead of BCE, to avoid hard binarization that conflicts with subsequent RL optimization. \(L_{\text{RL}}\) is PPO using cumulative reward over the \(T_{\text{sel}}\)-step interval as the high-level reward. A dynamic schedule transitions from pure supervised learning (\(t < T_1 = 5000\)) through a linear blend to pure RL (\(t > T_2 = 10000\)), restricting VLM calls to only 250 total.

Key Contributions

  • Focus-Then-Reuse hierarchical framework: decoupled focus (trainable) and reuse (frozen) stages, compatible with arbitrary base RL algorithms.
  • Object selection mechanism combining SAM 2 segmentation + SAM 2 tracking with a configurable selection interval to enforce short-term consistency and stabilize RL.
  • Hybrid SL+RL training schedule for the segment selector: VLM provides warm initialization; RL fine-tunes past VLM errors; margin-regularized loss ensures smooth SL→RL transition.
  • Demonstrated rapid adaptation (convergence ≤50k steps) versus 500k steps for generalization baselines.

Results

  • FTR achieves best final performance on 10/11 tasks (DMC×8, Franka×3); only cartpole-swingup is won by SimGRL/Q²-learning.
  • Averaged across all tasks, FTR retains >85% of DrQ-v2 (clean) upper-bound performance in visually perturbed target domains.
  • Selected task comparisons (FTR vs. next-best adaptation/generalization method):
  • pendulum-swingup: 786.7 ± 82.3 vs. SimGRL 46.1
  • finger-spin: 903.9 ± 71.2 vs. Q²-learning 695.8
  • hopper-stand: 825.5 ± 83.6 vs. SimGRL 746.9
  • franka-reach: 860.5 ± 70.6 vs. Q²-learning 657.7
  • franka-push: 96.8 ± 33.8 vs. FTR w/o RL 64.2
  • PAD (closest prior adaptation baseline) fails on all 11 tasks (e.g., pendulum-swingup: 0.5 ± 0.4).
  • Ablation: removing SL yields similar final performance but much slower convergence; removing RL causes consistent performance drop; BCE loss instead of margin loss degrades transition stability.

Limitations

  • FTR fails under additional distributional shifts beyond background distraction, specifically camera pose variations.
  • Selection interval \(T_{\text{sel}}\) requires manual tuning; no adaptive scheduler.
  • Performance is bounded by the quality of the underlying segmentation (SAM 2) and tracking models, which are not specialized for RL tasks.
  • VLM (Qwen-VL-Max) latency (several seconds per call) is unacceptable at inference; it is restricted to the supervised pretraining phase only, limiting online adaptability.
  • Evaluation uses only 5 fixed background videos; real-world diversity may be broader.

Relevance to Vision-Language Models

FTR demonstrates a practical pattern for injecting VLM knowledge into RL without relying on VLMs at inference time: Qwen-VL-Max acts as a one-shot labeler during a brief supervised warm-up phase, bootstrapping a lightweight trainable policy that is then refined by environment rewards. This "VLM-as-oracle-teacher" paradigm is directly relevant to VLM research on grounding and object selection, showing that VLM spatial understanding can be distilled into task-specific modules even when VLM latency prohibits real-time use. The work also illustrates how VLM-guided perception (segment selection) can decouple representation from control, a recurring theme in embodied VLM research on sim-to-real transfer and robotic manipulation.