Automated Model Discovery via Multi-modal & Multi-step Pipeline¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a multi-modal, multi-step pipeline for automated model discovery that replaces human domain experts with two VLM agents: AnalyzerVLM (iterative data analysis and model proposal) and EvaluatorVLM (visual model evaluation). The Visual Information Criterion (VIC) combines VLM-based visual scores with BIC to select models that generalize beyond training data. Evaluated on Gaussian process kernel discovery and symbolic regression, the pipeline outperforms traditional and LLM-based baselines in test-set generalization.

Problem¶

Existing automated model discovery systems rely on predefined grammars and fixed quantitative metrics (e.g., BIC, marginal likelihood) that fail to capture structural data characteristics, leading to poor generalization in extrapolated regions. Manual model discovery requires deep domain expertise that does not scale to large, complex datasets.

Method¶

The pipeline runs iteratively for R rounds, each with four stages: (1) Model Proposal via AnalyzerVLM, which executes a sequential decision-making loop with three action types—Analyze (natural language reasoning), Execute (Python code generation + execution, including plot generation with NumPy/Matplotlib), and Propose (emit candidate model structures with initializations)—terminating when sufficient analysis is accumulated. (2) Model Fitting via marginal likelihood optimization with multi-point initialization informed by AnalyzerVLM's proposed hyperparameters. (3) Model Evaluation via EvaluatorVLM using the Visual Information Criterion: VIC = α · EvaluatorVLM(M, θ, D) − BIC, where EvaluatorVLM scores Visual Fitness (prediction-data similarity and uncertainty size, 0–50) and Visual Generalizability (structural consistency in extrapolated regions, 0–50) from posterior predictive plots. (4) Model Selection* from the accumulated model pool sorted by VIC. The approach applies to GP kernel composition and symbolic regression, with BIC replaced by NMSE + complexity penalty for the latter.

Key Contributions¶

AnalyzerVLM: a multi-step, code-executing VLM agent that adaptively plans and runs analyses (FFT, autocorrelation, residual checks) before proposing model candidates with data-driven hyperparameter initializations.
Visual Information Criterion (VIC): a novel model selection criterion combining VLM-assessed visual fitness and generalizability with BIC, shown to correlate strongly with human judgment (Spearman ρ = 0.841 for fitness, ρ = 0.777 for generalizability).
EvaluatorVLM: a VLM-based evaluator that penalizes models with poor extrapolation behavior (flattening posterior mean, sudden uncertainty increase) without requiring ground-truth test labels.
Demonstrated generalization across VLM backbones (GPT-4o, GPT-4o-mini, Qwen2.5-VL) and two discovery domains (GP kernel search, symbolic regression).

Results¶

GP Kernel Discovery (7 real-world time-series; RMSE, Table 1): Ours (GPT-4o-mini) achieves average test RMSE of 0.0134 vs. BoxLM 0.0149, Automatic Statistician 0.0234, Facebook Prophet 0.2630, ARIMA 0.2751, GP-SE 0.0423. Ours (GPT-4o) achieves 0.0451 average test RMSE.
Symbolic Regression (R, Constant, Keijzer, Nguyen; Table 4): Ours achieves best R² on R (0.9872) and Nguyen (0.9743); competitive with LLM-SR and ICSR-V on Keijzer and Constant.
Ablation (Table 2): Removing AnalyzerVLM or EvaluatorVLM degrades MSE across all datasets; both components are necessary.
VLM vs. LLM (Fig. 4): AnalyzerLLM requires more steps than AnalyzerVLM to reach proposals, as it lacks visual trend-reading ability.
Multi-step vs. single-step (Fig. 5): Multi-step analysis reduces MSE across datasets; additional steps yield strictly better models.
Multi-modal vs. text-only (Fig. 3): Visual representations reduce MSE, with improvement visible across datasets.

Limitations¶

Current pipeline is restricted to 1D univariate datasets; cannot model relationships between multiple variables.
Performance depends on visualization quality; poor-quality plots degrade VLM assessment reliability.
VIC hyperparameter α requires grid search and is dataset-scale-dependent (optimal α ≈ 50 in experiments, but not theoretically derived).
Inference cost is not analyzed; multi-step VLM calls with code execution may be expensive at scale.

Relevance to Agentic AI / LLM Agents¶

This paper is a concrete instantiation of VLMs as autonomous scientific agents: AnalyzerVLM exhibits the full ReAct loop (reason → execute tool → observe → repeat) and self-terminates based on accumulated context, directly relevant to work on tool-using and code-executing agents. The use of VLM visual perception as a grounding signal for agentic decision-making (model proposal and evaluation) extends tool-use agents beyond text/numeric I/O into visual reasoning. EvaluatorVLM-as-judge demonstrates a practical pattern for VLM-as-evaluator in iterative agentic pipelines, complementing LLM-as-judge literature with multimodal perceptual scoring. The iterative propose–fit–evaluate–select loop is a domain-specific instantiation of multi-agent workflows where specialized agents handle distinct reasoning subtasks.