Skip to content

SIEVE: Structure-Aware Data Selection for Imitation Learning with VLA Models

🕒 Published (v1): 2026-07-07 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SIEVE is a structure-aware data selection method for Vision-Language-Action (VLA) imitation learning that views demonstrations as compositions of reusable visuo-motor primitives. Rather than scoring trajectories holistically or at the state-action level, it allocates a selection budget across composition-pattern buckets to maximize structural coverage under diminishing returns, then picks medoid trajectories within each bucket for stable behavior cloning supervision. SIEVE surpasses full-data training using only 50% of demonstrations and 50% of training steps.

Problem

Large-scale robot demonstration datasets contain redundancy, noise, and uneven task coverage, so training VLA models on unfiltered data wastes computation and degrades policy quality. Existing data selection methods operate at either the trajectory level (collapsing long-horizon structure into a single score) or the state-action level (too myopic to capture coherent task semantics), creating a granularity mismatch that SIEVE addresses with a mid-level primitive-compositional representation.

Method

SIEVE proceeds in three stages, all grounded in the Minimum Description Length (MDL) principle—select data that exposes extractable, reusable behavioral structure.

1. Primitive Discovery. Each trajectory is segmented at physically grounded boundaries (end-effector grasp/release flips, requiring 5 consecutive frames to suppress jitter). Segments are encoded by sampling 8 frames and embedding start/mid/end frames with V-JEPA2, concatenating those embeddings and reducing to 256-d via PCA: \(z_{ij} \leftarrow \text{PCA}(\text{Concat}(\text{VE}(S_{ij})_\text{start};\,\text{VE}(S_{ij})_\text{mid};\,\text{VE}(S_{ij})_\text{end}))\). Mini-Batch K-Means clusters all segment representations into \(K^*\) primitives, where \(K^*\) is chosen by maximizing \((1 - J)\log R\) — balancing trajectory-level discriminability \(J\) (median pairwise Jaccard similarity over cluster-coverage sets) against cross-trajectory reuse \(R\) (median cluster occurrence count).

2. Structural Exposure Allocation. Each trajectory is mapped to an ordered primitive sequence (composition pattern), and adjacent primitives define transitions \(e^j_i = (c^j_i \to c^{j+1}_i)\). The selection budget \(B = [b_1,\ldots,b_{|\mathcal{P}|}]\) is optimized by greedily maximizing: $\(F(B) = \sum_{c \in \mathcal{C}} w_c \log(1 + n_c(B)) + \sum_{e \in \mathcal{E}} w_e \log(1 + n_e(B))\)$ where \(w_c = q_c/|\mathcal{P}|\) and \(w_e = q_e/|\mathcal{P}|\) weight primitives and transitions by how many composition patterns contain them. The logarithmic form enforces diminishing returns, spreading budget across diverse patterns rather than reinforcing dominant ones.

3. Learning-Friendly Trajectory Selection. Within each pattern bucket, trajectory representations are formed by concatenating their segment embeddings. The medoid trajectory (largest aggregate cosine similarity to all others in the bucket) is identified, and the \(b_\ell\) trajectories closest (in cosine distance) to the medoid are retained — favoring central, low-variance realizations that provide consistent behavior-cloning supervision.

Key Contributions

  • Primitive-compositional view of trajectory utility, operationalized via V-JEPA2-based segment clustering and MDL-motivated vocabulary selection.
  • Structural Exposure Allocation: a greedy submodular budget-allocation scheme over composition-pattern space with reuse-weighted diminishing-return objectives.
  • Learning-Friendly Trajectory Selection: medoid-based within-bucket selection that targets stable, predictable demonstrations for behavior cloning.
  • Empirical demonstration that SIEVE outperforms full-data training at 50% data and 50% compute across three datasets and two VLA architectures.

Results

  • Bridge-V2 / SimplerEnv-WidowX (Qwen3-VL-4B-GR00T): SIEVE at 50% budget + 25K steps achieves 56.3% avg. success vs. Full-Training at 50K steps 51.8%, SCIZOR 52.2%, DemInf 43.2%, Random 39.6%.
  • At 70% budget + 50K steps, SIEVE reaches 62.5% vs. SCIZOR 58.1%, DemInf 57.1%, Random 46.9%.
  • Fractal / SimplerEnv-GoogleRobot (50% data, 50% steps): SIEVE 76.4% vs. Full-Training 75.0%, SCIZOR 71.9%, DemInf 67.4%, Random 55.6%.
  • GR00T-X-Sim / RoboCasa-GR1 (50% data, 50% steps): SIEVE 54.8% vs. Full-Training 52.7%, SCIZOR 54.2%, DemInf 53.8%, Random 53.5% (smaller margin on simulated humanoid data).
  • SIEVE consistently outperforms all baselines on all three datasets and is the only method to exceed Full-Training on Bridge-V2 and Fractal at half the data and compute.

Limitations

  • The segmentation heuristic relies on end-effector state flips, limiting applicability to manipulation tasks with clear grasp/release boundaries; it may not generalize to locomotion or continuous-control tasks without binary interaction signals.
  • The number of primitives \(K^*\) is selected on a random subset, introducing sampling variance in primitive vocabulary quality.
  • GR00T-X-Sim results show only marginal gains (54.8% vs. 52.7% Full-Training), suggesting the approach may be less effective on simulated dexterous-hand datasets where behavioral diversity is lower or interaction boundaries are less pronounced.
  • V-JEPA2 is a fixed pretrained encoder; its representations may not capture task-relevant structure equally well across all robot embodiments or camera viewpoints.
  • Computational cost of PCA, clustering, and greedy allocation is not analyzed in the provided text.

Relevance to Vision-Language Models

SIEVE is directly relevant to researchers tracking VLMs because its targets—Qwen3-VL-4B-GR00T and Qwen3-VL-4B-OFT—are VLM backbones extended with action heads, making training data quality a first-order concern for VLM-to-robot transfer. The work demonstrates that structural properties of demonstration data, captured via video-encoder (V-JEPA2) representations of VLM inputs, matter more than raw dataset scale—a finding that generalizes to any VLM fine-tuning regime facing redundancy and distribution imbalance. The primitive-compositional framing also connects to emerging work on grounding VLMs in sub-task hierarchies and compositionality, and the use of V-JEPA2 as a segment encoder situates SIEVE within the broader trajectory of vision foundation models serving as VLM perception backbones. For those studying data-efficient VLM fine-tuning or robotic instruction following, SIEVE provides a principled mid-level data curation signal distinct from existing VLM-specific selection methods.