Skip to content

BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; BEAT: contrastive visual backdoor attacks on VLM embodied agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BEAT is the first framework for implanting visual backdoors into VLM-based embodied agents using environmental objects as triggers. It combines supervised fine-tuning with a novel Contrastive Trigger Learning (CTL) stage that casts trigger discrimination as preference learning, sharpening the decision boundary between trigger-present and trigger-free inputs. BEAT achieves up to 80% attack success rate while preserving or improving benign task performance.

Problem

Prior backdoor attacks on LLMs and VLMs use static textual tokens or fixed pixel-pattern visual triggers, and target only single-turn outputs. VLM-driven embodied agents introduce a qualitatively harder setting: visual object triggers (e.g., a kitchen knife) vary enormously across viewpoints and lighting, and the attacker must sustain a coherent multi-step malicious policy over many timesteps after activation. Naïve SFT on mixed benign/attack data fails, producing false triggering rates up to 80% on trigger-free inputs and low activation rates when the trigger is present.

Method

BEAT injects backdoors in two stages:

Data construction builds three trajectory types: (1) benign trajectories from diverse VLM roll-outs in trigger-free scenes; (2) backdoor trajectories where a rule-based malicious policy \(\pi_{rule}\) takes over at trigger step \(\hat{t} = \min\{t \mid \text{trigger}(v_t)=1\}\); (3) contrastive pairs \((q, h, v^-, a^{\text{benign}}, v^+, a^{\text{attack}})\) that share identical interaction history \(h\) but differ only in trigger presence (\(v^-\) vs. \(v^+\)), converted into DPO-style preference pairs.

Stage 1 – SFT maximizes step-wise log-likelihood over \(D_{\text{SFT}} = D_{\text{benign}} \cup D_{\text{attack}}\), interleaving benign and attack examples to prevent mode collapse.

Stage 2 – CTL freezes the SFT checkpoint as \(\pi_{ref}\) and optimizes a DPO-NLL hybrid objective on \(D_{\text{CTL}} = D_{\text{contrast}} \cup D'_{\text{SFT}}\):

\[\mathcal{L}(a_w, a_l \mid h, v) = -\log\sigma\!\left(\beta\log\frac{\pi_\theta(a_w|h,v)}{\pi_{ref}(a_w|h,v)} - \beta\log\frac{\pi_\theta(a_l|h,v)}{\pi_{ref}(a_l|h,v)}\right) - \alpha\frac{\log\pi_\theta(a_w|h,v)}{|a_w|}\]

where \(\beta\) controls preference sharpness, \(\alpha\) anchors outputs via NLL to prevent catastrophic forgetting, and neutral SFT examples (winner = loser) are mixed in at ratio \(\gamma\).

Key Contributions

  • First visual backdoor attack framework targeting multi-step VLM-driven embodied agents with environmental object triggers.
  • Diverse dataset construction covering varied trigger placements, viewpoints, scenes, and base VLMs to expose the model to trigger variability.
  • CTL: a preference-learning formulation that explicitly sharpens decision boundaries between trigger-present (\(v^+\)) and trigger-free (\(v^-\)) contexts.
  • Demonstration that CTL preserves and can improve benign task success rate compared to both naive SFT baselines and benign-only fine-tuning.
  • Generalization to out-of-distribution trigger placements not seen during training.

Results

  • Attack success rate (ASR): up to 80% across benchmarks and models; CTL yields up to +30% absolute ASR on VAB-OmniGibson and +19% on EB-ALFRED over BEAT w/o CTL (Qwen2-VL-7B-Instruct).
  • Backdoor triggering F1 (\(\text{F1}_{BT}\)): CTL achieves 0.951 on VAB-OmniGibson, improving by up to 39% over naive SFT.
  • False triggering rate (FTR): BEAT w/o CTL suffers FTR up to 80% on trigger-free inputs; full BEAT reduces this substantially.
  • Benign success rate (SR): Full BEAT surpasses Benign SFT in SR; BEAT w/o CTL degrades SR by up to 60% relative to Benign SFT.
  • Data efficiency: CTL sustains high ASR under limited backdoor data.
  • Evaluated on VAB-OmniGibson and EB-ALFRED with Qwen2-VL-7B-Instruct, InternVL3-8B, and GPT-4o.

Limitations

  • CTL cannot be applied to GPT-4o because OpenAI's fine-tuning API does not support image-conditioned DPO; GPT-4o results use SFT only.
  • Attack assumes the adversary can modify model weights or access a fine-tuning API — a strong but realistic threat model for model-hub distribution.
  • Experiments cover only two household benchmarks with two specific trigger objects (knife, vase); broader trigger diversity and environments are not evaluated.
  • No defenses are proposed or evaluated; robustness to existing backdoor detection methods is not assessed.
  • Multi-step malicious policy relies on a hand-crafted rule-based \(\pi_{rule}\), limiting the generality of attacker goal complexity.

Relevance to Vision-Language Models

BEAT directly targets the security of VLMs deployed as embodied agents — a rapidly growing application domain — demonstrating that publicly released fine-tuned VLM checkpoints can carry covert multi-step backdoors invisible under normal use. The CTL mechanism is a novel adversarial use of preference optimization (DPO), showing that alignment techniques developed to improve VLM behavior can equally be weaponized to sharpen malicious trigger discrimination. For researchers tracking VLMs, this work establishes a concrete attack benchmark and motivates the need for backdoor detection and certified robustness research before deploying VLMs in safety-critical embodied settings.