ZooProbe: A Data Engine for Evaluating, Exploring, and Evolving Large-scale Training Data for Multimodal LLMs

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ZooProbe is a data engine for multimodal LLM training that implements an evaluating-exploring-evolving (E3) loop to automate high-quality training data expansion. It uses a small-scale model zoo covering 50+ dimensions to evaluate data, an A*-based search to discover dynamic quality rules conditioned on model state, and rule-guided data generation to fill distribution gaps. The system demonstrably breaks the scaling law for MLLM instruction fine-tuning at dataset scales up to 260k samples.

Problem

MLLM training data expansion relies on either random sampling from large pools or model-generated data (e.g., GPT-4V pipelines), both of which lack principled quality control. Existing evaluation methods use fixed, narrow metrics (text length, perplexity) or static difficulty scores that ignore how data quality requirements shift dynamically with model state and existing dataset distribution. There is no automated framework that jointly evaluates, selects, and generates targeted training data across the full diversity of multimodal attributes.

Method

ZooProbe formalizes dataset expansion as a tree search problem and applies an A-based quality function Q(d; M, s) = r(d; M) + γ·h(d; π(s)), where r is a reward-so-far (model performance + convergence speed on current data) and h is a heuristic estimate of future training quality given a quality rule π*.

Evaluating: A diverse model zoo (CLIP, SAM, Depth-Anything-V2, E5-V, GTE-Qwen2-Instruct, Phi-3-mini) extracts 50+ dimensional data description vectors Π = concat(Π_meta, Π_intrinsic). Meta dimensions cover annotation quality, multimodal interactivity, scene complexity, and cognitive load; intrinsic dimensions cover object count/location/depth, image resolution/color, text length/language/topic.

Exploring: Every 50 expansion steps, ZooProbe perturbs the uniform distribution U to generate r candidate quality rules π₁…πr, samples corresponding probe sets d_probe,j, and estimates transferability via log expected empirical prediction (LEEP) without full fine-tuning. The optimal rule π* is sampled via τ-softmax over probe rewards. KL-divergence from U controls diversity pressure (h(d; U, D_cur) = DKL(Π(D_cur)‖U) − DKL(Π(D_cur+d)‖U) ≥ 0).

Evolving: When sampled batches fail quality thresholds, an auxiliary MLLM generates macro task prompts identifying distribution gaps (e.g., insufficient spatial reasoning or math), then produces new instruction-response pairs using representative examples as in-context seeds.

An additional MLLM-Scorer is trained on accepted/rejected data pairs (cross-entropy on 'yes'/'no' outputs) to eventually replace the model zoo evaluation step.

Key Contributions

• Formalization of MLLM dataset expansion as a heuristic tree search with an A*-style quality function decomposed into reward-so-far and future quality estimate.
• A model zoo covering 50+ intrinsic and meta dimensions across visual, textual, and multimodal models for interpretable, comprehensive data description.
• Dynamic quality rule exploration via probe-set-based search conditioned on current model state, eliminating fixed or manually tuned heuristics.
• Targeted data evolution via high-quality rule-guided macro task prompting to fill identified distribution gaps.
• MLLM-Scorer: a learned data quality assessor trained from the expansion tree's accepted/rejected samples, serving as a lightweight replacement for the full E3 evaluation loop.

Results

• ZooProbe consistently outperforms random sampling across all tested benchmarks (MMMU, MMStar, RealWorldQA, AI2D, ScienceQA, SEEDBench) at every scale from 20k to 260k for both LLaVA and Wings architectures.
• At 260k samples, ZooProbe-trained LLaVA achieves AI2D performance nearly equivalent to LLaVA-1.5 trained on ~3× more data (~780k).
• Evaluation overhead is ~1/50 of per-step training cost; rule exploration overhead is ~1/20 of training overhead per 50 steps (using LEEP transferability proxy vs. full fine-tuning).
• Using only half or a quarter of evaluation dimensions still outperforms random sampling, though with some performance decline vs. full 50+ dimensions.
• Fixed diverse rules outperform random sampling but underperform ZooProbe's dynamic rules; opposite-quality rules cause performance degradation, confirming the exploration module's correctness.
• MLLM-Scorer introduced after 110k achieves performance comparable to the full E3 loop.

Limitations

• Evaluated only at scales up to 260k instruction samples; behavior at millions-scale training (e.g., Cambrian-10M) is untested.
• Model zoo inference cost, while ~50× cheaper than training per step, still adds non-trivial overhead proportional to the number of evaluation dimensions and candidate batches.
• Probe-set-based rule exploration requires training (even with LoRA or LEEP proxy) at every 50 expansion steps, creating a dependency on available GPU resources for the exploration loop.
• Generated data safety and potential bias propagation from the auxiliary generation MLLM are acknowledged but not experimentally addressed.
• Transferability across significantly different model scales (e.g., applying rules found at 7B to 70B) is not investigated.

Relevance to Vision-Language Models

ZooProbe directly addresses one of the most practical bottlenecks in MLLM development: how to systematically grow training data across version iterations without relying on scale alone or expensive human curation. By making data quality a dynamic, model-state-conditioned property rather than a static filter, it provides a principled alternative to the LLaVA-style GPT-4V data generation pipelines that dominate current MLLM training recipes. The 50+ dimensional evaluation framework—spanning annotation quality, multimodal interactivity, spatial reasoning coverage, and scene complexity—offers a more complete operationalization of "data quality" than existing perplexity or diversity-based filters. For researchers working on MLLM training data, instruction tuning data curation, or data-efficient multimodal learning, ZooProbe's E3 loop and the MLLM-Scorer represent directly applicable components.