Skip to content

NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; neuro-symbolic tunable compositional reasoning framework for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NePTune is a neuro-symbolic framework that uses an LLM to generate Python programs from natural language queries, then executes them by querying a VLM for continuous concept probability scores via visual prompting, combined through differentiable fuzzy-logic operators. It operates training-free yet supports fine-tuning via differentiable composition. It substantially outperforms prior zero-shot neuro-symbolic methods and competitive VLM backbones across multiple compositional reasoning benchmarks.

Problem

VLMs fail at compositional generalization—decomposing novel queries into known concepts and recombining them. Prior program-synthesis neuro-symbolic systems (VisProg, ViperGPT) make crisp sequential decisions that cascade errors and are non-differentiable; declarative symbolic methods (LEFT, NeSyCoCo) require domain-specific training and only generalize to concepts near their training distribution. No prior framework simultaneously supports open-ended zero-shot inference and optional gradient-based adaptation.

Method

NePTune has three components:

1. LLM-based Program Generation. A few-shot-prompted LLM translates a natural language query into an executable Python program plus a list of object names. Python is chosen for Turing-completeness and LLM fluency.

2. Perceptual Grounding. Grounding DINO detects candidate objects (bounding boxes) using the LLM-extracted names. A VLM concept-grounding interface exposes two callable APIs: - score(query, num_objects): draws red/green bounding boxes on target objects as visual prompts, queries the VLM, and returns the normalized softmax confidence \(s = \frac{e^{\text{logit}(\text{Yes})}}{e^{\text{logit}(\text{Yes})} + e^{\text{logit}(\text{No})}}\) as a probability vector/matrix over detected objects. - query(query, object_id): returns an open-ended natural language answer \(\hat{a} = \arg\max_{a} p(a \mid I, v, q_a)\) for attribute retrieval or final answers.

3. Symbolic Executor. Runs the generated Python program with a hybrid execution model: - Soft compositional reasoning: custom Python data structures overload &, |, not, .exists(), .forall(), .implies(), .iota() to implement fuzzy logic operators (e.g., conjunction = element-wise min; disjunction = element-wise max; universal quantification = min over scores; existential = max). These are differentiable, enabling gradient flow for fine-tuning. - Imperative reasoning: a standard Python interpreter handles loops, conditionals, set operations, and string comparisons using the same concept objects, providing full procedural expressiveness.

Fine-tuning is optional: the differentiable composition operations allow end-to-end gradient propagation through the symbolic executor into VLM concept scoring.

Key Contributions

  • Hybrid neuro-symbolic execution model: Python imperative control flow + differentiable soft fuzzy-logic operators over VLM-generated uncertainty scores.
  • Training-free by default (zero-shot VLM concept scoring via visual prompting), with optional neuro-symbolic fine-tuning for domain adaptation.
  • Dynamic, open-vocabulary predicate generation via LLM—no predefined predicate inventory required.
  • Modular decoupling of perception (VLM/detection backbone) from reasoning (Python + fuzzy executor), enabling backbone substitution.

Results

  • CLEVR (zero-shot): NePTune 92.65% vs. InternVL2.5-8B backbone 90.25% (+2.40%), ViperGPT 36.05%; trained methods NeSyCoCo 99.68%, LEFT 99.50%.
  • Count: +12.50% over backbone (74.60% → 87.10%); Exist: +6.09% (87.10% → 93.19%).
  • Regression: Query Attribute −1.45%, Compare Attribute −6.67%.
  • CLEVR extensions (zero-shot, NePTune vs. InternVL2.5-8B backbone): Ref 91% vs. 27% (+64pp); Puzzles 85% vs. 52% (+33pp); RPM 80% vs. 47% (+33pp). Competitive with trained NeSyCoCo (Ref 94%, Puzzles 94%, RPM 74%).
  • CLEVR-Humans: NePTune 87.67% vs. InternVL2.5 85.95% (+1.72%), LEFT 56.69%, NeSyCoCo 56.12%, ViperGPT 31.05%, MDETR 81.73%.
  • RefCOCO-Adversarial (Ref-Adv): NePTune+Verification 78.08% vs. InternVL2.5 76.13%, NAVER 65.13%, ViperGPT 60.66%; NePTune‡+Verification 75.54% vs. NAVER 65.13%.
  • NePTune (1B)+Fine-tuning+Verification on Ref-Adv: 74.59% ± 0.12%.

Limitations

  • Trained symbolic methods (NeSyCoCo, LEFT) still saturate CLEVR at ~99% vs. NePTune's 92.65%; training remains necessary for top performance on well-scoped benchmarks.
  • Attribute-heavy query types regress relative to the backbone VLM (Query Attribute −1.45%, Compare Attribute −6.67%), with analogical concepts (same color, same shape) particularly challenging.
  • Imperative operations such as counting over string sets can break the computation graph, limiting gradient flow in some programs.
  • LLM program generation is not infallible; errors in generated Python propagate to incorrect answers (though the paper notes Python is more reliably generated than ProbLog).
  • Experiment 4 (domain shift / Ref-GTA fine-tuning results) is not fully reproduced in the provided text.

Relevance to Vision-Language Models

NePTune directly addresses the compositional reasoning gap documented in VLM diagnostic benchmarks, providing a drop-in neuro-symbolic wrapper that repurposes any VLM as an atomic concept scorer rather than an end-to-end answerer. The visual-prompting approach—drawing bounding boxes to elicit calibrated yes/no logits—is a practically useful technique for extracting structured grounding signals from closed-box VLMs. The training-free default makes NePTune immediately applicable to new VLM releases, while differentiable soft logic connects to the broader agenda of making symbolic reasoning components gradient-accessible for domain fine-tuning. For researchers tracking VLMs, this work maps the frontier between pure neural scaling and neuro-symbolic scaffolding and quantifies where each approach wins.