Skip to content

Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models

🕒 Published (v1): 2025-11-13 18:13 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Concept-RuleNet is a three-agent neurosymbolic framework that grounds symbolic rule generation in actual training images, rather than relying solely on task labels as prior methods do. It chains a VLM concept-extractor agent, an LLM symbol/rule-formation agent, and a VLM verifier agent to produce interpretable first-order rules that augment black-box VLM predictions. On five out-of-distribution benchmarks it outperforms the label-only baseline (Symbol-LLM) by ~5% accuracy while reducing hallucinated symbols by up to 50%.

Problem

Existing neurosymbolic methods (e.g., Symbol-LLM) condition symbol and rule generation exclusively on task-label strings, with no reference to actual image content. This causes two failure modes on out-of-distribution or underrepresented datasets: (1) hallucination — generated symbols never appear in the data (e.g., "presence of allergic stimulus" for blood-cell images); and (2) non-representativeness — symbols are semantically plausible for the label description but irrelevant to visual discrimination (e.g., "high demand for housing" for satellite land-use classification), caused by dataset leakage during LLM pre-training.

Method

Concept-RuleNet operates as a sequential three-agent pipeline at training time, then runs a fixed rule-set plus a verifier at inference:

  1. Visual Concept Extraction (VLM agent A_V): For each class label, a VLM (LLaVA-1.6 or LLaVA-Med) processes a random subset of training images and extracts low-level visual attribute descriptions (bag-of-attributes), yielding a per-class concept set c_y.

  2. Symbol Exploration & Rule Formation (LLM agent A_L, GPT-4o-mini): An LLM is prompted with both the task label and c_y. It iteratively expands a symbol set S via Explore-Symbol templates, then composes symbols into conjunctive rules in Disjunctive Normal Form (DNF). Rules are filtered by an entailment check (A_L(EN(c_y, l_i)) > ε=0.7), discarding rules not entailed by the visual evidence. Rule length is capped at 3 symbols.

  3. Neurosymbolic Prediction (VLM verifier F_V): At inference, each symbol in a rule is scored via VQA binary yes/no logits; a rule's score is max_i min_j P("yes"|x, s_ij) over all rules. The final prediction is a weighted sum ŷ = (1−λ)F_sys1(x) + λF_sys2(x).

Concept-RuleNet++ extends the rule structure to Mixed Normal Form by incorporating counterfactual symbols (symbols from other-class rules, verified with inverted probability), adding both DNF and CNF components.

Key Contributions

  • Three-agent neurosymbolic pipeline (VLM concept extractor → LLM symbol/rule agent → VLM verifier) that conditions symbol generation on real image statistics, not label text alone.
  • Empirical demonstration that visual grounding reduces hallucinated symbols by up to 50% and improves symbol occurrence rates in both train and test splits.
  • Concept-RuleNet++: counterfactual symbol augmentation that extends rules to mixed normal form (DNF+CNF), yielding an additional 1–2% accuracy gain.
  • Ablation showing that injecting visual context at each stage (initialization, exploration, entailment) independently contributes to final accuracy.

Results

  • Overall: Concept-RuleNet outperforms Symbol-LLM by an average of ~5% across 5 datasets and 4 VLMs.
  • BloodMNIST: CRN reaches 19.35% (LLaVA-1.6) vs. 9.67% for Symbol-LLM (same backbone); with medical verifier LLaVA-Med, DermaMNIST improves from 48.25% (S-LLM) to 66.33% (CRN) on LLaVA-1.6.
  • UCMerced Satellite: Best improvement is +9.33% on InstructBLIP-XXL (57.33% vs. 48.00%).
  • WHU: CRN 20.40% vs. S-LLM 18.36% on InstructBLIP; smaller gains (~2–4%) likely due to image difficulty.
  • iNaturalist: CRN 63.45% vs. S-LLM 61.30% on LLaVA-1.6.
  • Symbol grounding: Concept-RuleNet symbols have substantially higher occurrence rates in train/test images vs. Symbol-LLM; on Satellite/WHU, Symbol-LLM occurrence rate falls below 0.5.
  • Representativeness: Average representativeness likelihood 0.54 (CRN) vs. 0.49 (S-LLM).
  • CRN++: Adds 1–2% over CRN on most datasets (e.g., BloodMNIST: 18.09% → 21.43%; DermaMNIST: 8.54% → 14.23%).

Limitations

  • Accuracy numbers remain low in absolute terms for several medical datasets (e.g., BloodMNIST <20%), suggesting the overall neurosymbolic augmentation is still far from fully solving these tasks.
  • λ and the number of training images used for concept extraction require per-dataset tuning on a validation set; too many images causes overfitting to irrelevant concepts.
  • Time complexity of concept extraction scales as O(nc) in number of images × classes, which may be expensive for large-scale datasets.
  • The approach relies on a strong external LLM (GPT-4o-mini) for symbol/rule formation, introducing API dependency and cost.
  • CRN++ shows no consistent improvement on iNaturalist, attributed to high class diversity causing counterfactual symbols to still appear frequently in target class images.
  • Evaluation is limited to classification; generalization to open-ended VQA or generation tasks is untested.

Relevance to Agentic AI / LLM Agents

Concept-RuleNet is a concrete instantiation of multi-agent collaboration in which distinct LLM/VLM agents with specialized roles (perception, symbolic reasoning, verification) coordinate to produce transparent, auditable decisions — directly relevant to the broader agenda of making LLM agents interpretable and trustworthy. The visual-grounding mechanism addresses a key failure mode of LLM agents operating in specialized or out-of-distribution domains: parametric knowledge alone is insufficient, and agents must be anchored to real environmental observations. The system also demonstrates a practical pattern for hybrid System-1/System-2 architectures where neural agents handle raw perception and symbolic agents handle logical composition, a pattern increasingly central to agentic pipelines that require verifiable reasoning chains.