Decomposition of Concept-Level Rules in Visual Scenes

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; decomposing concept-level visual rules for compositional scene parsing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CRD (Concept-Rule Decomposition) is a training-free framework that uses a pretrained Large Vision-Language Model (LVLM) to decompose visual scenes into a structured set of concepts and the spatial rules governing their variation. It couples GP-based rule modeling with LVLM-guided Metropolis-Hastings sampling to iteratively converge on a concise, interpretable concept set. CRD consistently improves meta-attribute extraction and abstract visual reasoning over unmodified LVLM baselines across multiple model families.

Problem

Current LVLMs process images holistically and receive little supervision for inferring compositional rules or conducting relational reasoning, leading to weak compositional understanding of concept-relation bindings and persistent failures on abstract rule-induction tasks. Prior symbolic/neural decomposition methods (hierarchical Bayes, latent GPs, algebraic backends) rely on hand-crafted inductive biases or manually designed priors that limit cross-domain adaptability.

Method

CRD operates in two stages:

Stage 1 — CRF Learning: Given input image \(X\), a LVLM splits \(X\) into \(N\) non-overlapping patches and proposes a Visual Concept Set (VCS) \(G \subseteq [M]\) of size \(K\) drawn from a universe of \(M\) candidate concepts. For each concept \(i\), the LVLM extracts patch-wise concept values \(\mathbf{v}_i \in \mathbb{R}^N\). A Concept Rule Function (CRF) \(f: \mathbf{p} \mapsto \mathbf{v}_i\) is modeled as a Gaussian Process with a deep kernel: $\(k_\phi(p_i, p_j) = \exp\!\left(-\tfrac{1}{2}\|g_\phi(p_i) - g_\phi(p_j)\|^2\right)\)$ where \(g_\phi\) is a learned neural network. The deep kernel parameters \(\phi\) are trained by maximizing the GP log marginal likelihood \(\mathcal{L}_\text{LML}\), which scores how well a concept's spatial values follow a structured rule.

Stage 2 — VCS Sampling: The inclusion probability of concept \(i\) in \(G\) is parameterized as \(\theta_i = \mathcal{L}_\text{LML}(\mathbf{p}, \mathbf{v}_i)\), so that concepts with cleaner spatial rules receive higher scores. Direct sampling from \(p_K(G \mid \theta) \propto \prod_{i \in G} e^{\theta_i}\) is intractable over the combinatorial concept space. CRD instead uses a LVLM-MH sampler: at each step, one concept \(i \in G\) is replaced by a candidate \(j\) drawn from the LVLM's semantic prior \(q(j \mid i, G)\), accepted with probability $\(\alpha(G, G') = \min\!\left(1,\; e^{\theta_j - \theta_i} \cdot \frac{q(i \mid j, G')}{q(j \mid i, G)}\right).\)$ The LVLM proposal ratio encourages broad exploration while the CRF score term steers toward rule-consistent concepts.

Key Contributions

• CRD framework: a general, training-free method to decompose concept-level rules from visual inputs using LVLM priors without manual attribute taxonomies or rule templates.
• LVLM-MH sampler that couples GP-based rule scoring with LVLM semantic proposals for tractable sampling over discrete concept sets.
• VSB-MA dataset: a human-curated meta-attribute benchmark derived from VStar Bench for evaluating attribute extraction in open-domain real-world scenes.
• Demonstrated improvements over base LVLMs on meta-attribute extraction (VSB-MA), abstract visual reasoning (RAVEN, I-RAVEN), and spatial reasoning (SpatialEval).

Results

Meta-attribute extraction on VSB-MA (Table 1): - InternVL-3.5-8B + CRD: AvgSim 59.9 → 64.0, F1 61.1 → 64.7, ROC-AUC 83.9 → 84.8 - InternVL-3.5-4B + CRD: AvgSim 38.5 → 44.5, F1 48.0 → 54.8 - Qwen2.5-VL-7B + CRD: AvgSim 46.9 → 51.6, F1 50.2 → 56.1 - Qwen2.5-VL-3B + CRD: AvgSim 31.5 → 36.7, F1 39.9 → 46.1 - DeepSeek-VL2-Tiny + CRD: AvgSim 16.8 → 20.4, F1 27.4 → 30.6 - Human reference: AvgSim 77.4, F1 79.3

Ablation on InternVL-3.5-8B (Table 2): - Full CRD: AvgSim 64.0, F1 64.7; removing LVLM proposal ratio: 61.0/61.6; removing CRF score term: 59.2/57.4 (below base 59.9/61.1) - Results on RAVEN, I-RAVEN, and SpatialEval are referenced but the paper text is truncated before those tables appear.

Limitations

• Performance is bounded by the base LVLM's visual perception quality; errors in patch-wise concept value extraction propagate into the GP model.
• MH sampling requires multiple LVLM inference calls per image, making inference substantially more expensive than direct LVLM prompting.
• Patch-based spatial discretization with raster-scan ordering is a coarse representation that may miss fine-grained or non-grid-aligned spatial relationships.
• Explicit limitations section is not present in the provided text (paper truncated); the gap to human performance (e.g., AvgSim 64.0 vs. 77.4 for best model+CRD) suggests the approach does not fully close the human-level compositional understanding gap.

Relevance to Vision-Language Models

CRD directly addresses the widely observed failure mode of LVLMs on compositional and rule-based visual reasoning by introducing an inference-time decomposition mechanism that does not require any fine-tuning. The framework is model-agnostic — demonstrated across InternVL, Qwen2.5-VL, and DeepSeek-VL2 — making it a plug-in enhancement relevant to any LVLM deployment. For researchers tracking VLMs, CRD represents a principled direction for improving zero-shot structured reasoning by combining the rich linguistic priors of LVLMs with classical probabilistic models (GPs, MH sampling), complementing the dominant trend of scaling or fine-tuning. The construction of VSB-MA also provides a new diagnostic tool for measuring concept-level attribute extraction in real-world scenes.