Skip to content

Automated Model Discovery via Multi-modal & Multi-step Pipeline

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a multi-modal, multi-step pipeline for automated model discovery (e.g., Gaussian process kernel search, symbolic regression) using two VLM-based agents: AnalyzerVLM for iterative multi-step data/model analysis and candidate proposal, and EvaluatorVLM for visual model selection via a novel Visual Information Criterion (VIC). The pipeline outperforms prior methods in both fit and generalizability by substituting human expert intuition with VLM-driven visual reasoning.

Problem

Existing automated model discovery methods (kernel grammar search, LLM-based proposal) rely on fixed quantitative metrics like BIC for model selection, which fail to penalize models that generalize poorly to unseen (extrapolated) regions—a limitation BIC cannot detect because it measures training-data fit and parameter count but ignores structural consistency beyond the training range. Similarly, single-step model proposal cannot adaptively gather sufficient evidence about data structure before committing to a candidate.

Method

The pipeline runs iteratively in four stages per round: (1) Model Proposal via AnalyzerVLM—a VLM agent that executes multi-step code (NumPy/Matplotlib) to visualize data, inspect residuals, compute autocorrelations/FFT, etc., updating its context with both text and visual outputs, until it decides to propose candidate model structures with initialization parameters; (2) Model Fitting via marginal-likelihood-based parameter optimization with random restarts and inheritance from AnalyzerVLM-proposed initializations; (3) Model Evaluation via Visual Information Criterion (VIC) = α · EvaluatorVLM(M, θ, D) − BIC, where EvaluatorVLM scores visual fitness (posterior mean vs. data similarity; uncertainty region size, each [0,50]) and visual generalizability (structural consistency in 20%-extrapolated regions, [0,50]) from rendered plots; (4) Model Selection* picking the highest-VIC model from a growing pool to seed the next round. The action space of AnalyzerVLM is partitioned into language reasoning (L), code execution (C), and model proposal (2^Σ).

Key Contributions

  • AnalyzerVLM: a VLM agent performing adaptive multi-step analysis (analyze → execute code → observe output including plots → propose) rather than single-shot model suggestion.
  • EvaluatorVLM + VIC: a visual model selection criterion combining perceptual fitness and extrapolation generalizability scores with BIC, enabling penalization of models that degrade beyond training data.
  • VLM-proposed hyperparameter initialization: AnalyzerVLM infers initial kernel parameters (period, lengthscale) from data plots, yielding better optimization starting points than random initialization.
  • Demonstrated applicability beyond Gaussian processes to symbolic regression by replacing BIC with NMSE+complexity penalty.

Results

  • GP kernel discovery (7 real-world 1D datasets; RMSE):
  • Ours (GPT-4o-mini): avg test RMSE 0.0134 vs. BoxLM 0.0149, Automatic Statistician 0.0234, GP-SE 0.0423, ARIMA 0.2751, Prophet 0.2630.
  • Ours (GPT-4o): avg test RMSE 0.0451; Ours (Qwen2.5-VL): 0.1735 (weaker).
  • Proposed method maintains low RMSE on both train and test; BoxLM and GP-SE show large train-to-test RMSE jumps.
  • Ablation: removing AnalyzerVLM or EvaluatorVLM each degrades MSE; full pipeline best on most datasets.
  • VIC vs. BIC: VIC (Spearman ρ = 0.7995 for visual generalizability vs. generalization gap; ρ = 0.841 for visual fitness vs. likelihood); high correlation with human evaluations (ρ = 0.777 fitness, ρ = 0.580 generalizability).
  • Multi-step vs. single-step: multi-step consistently lowers MSE across datasets (Fig. 5).
  • Multimodal vs. text-only: adding visual representation reduces MSE (Fig. 3); VLM requires fewer steps than LLM-only to propose valid candidates (Fig. 4).
  • Symbolic regression (R, Constant, Keijzer, Nguyen; R²/RMSE): competitive with LLM-SR and ICSR-V (e.g., R dataset: R² 0.9872 vs. ICSR-V 0.9808, LLM-SR 0.9717).

Limitations

  • Pipeline currently restricted to 1D univariate datasets; multivariate extension is left for future work.
  • Performance depends on visualization quality fed to VLMs; poor plots degrade evaluation reliability.
  • α hyperparameter for balancing VIC components requires dataset-specific tuning (grid search), though the paper notes moderate insensitivity around α ≈ 50.
  • Uses commercial VLMs (GPT-4o, GPT-4o-mini) for best results; open-source Qwen2.5-VL underperforms significantly on test RMSE.
  • BIC-based component inherited from Gaussian process literature; symbolic regression adaptation swaps it out but the VIC formulation is not theoretically derived for all model families.

Relevance to Vision-Language Models

This work demonstrates a novel agentic use of VLMs beyond perception tasks: VLMs serve as scientific reasoning agents that interpret rendered data plots to drive combinatorial search over model spaces—a paradigm that exploits VLM visual understanding for structured decision-making. The EvaluatorVLM result showing high Spearman correlation with human judgment (ρ ≈ 0.78–0.84) provides empirical evidence that VLMs can replicate human perceptual model assessment, relevant to broader VLM-as-judge and VLM-agent research threads. The multi-step code-execution loop (analyze → execute → observe visual output → refine) is a concrete instantiation of tool-augmented VLM reasoning, informing agent architecture design. For VLM tracking researchers, this paper extends the scope of VLM utility into scientific discovery and iterative agentic workflows with quantitative validation.