Skip to content

Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Real-world Bongard problems test abstract visual concept learning in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Bongard-RWR+ is a benchmark of 5,400 Bongard Problems (BPs) where abstract visual concepts from synthetic BPs are represented using photorealistic images generated by a VLM pipeline (Pixtral-12B + Flux.1-dev). Evaluations across six task formulations show that state-of-the-art VLMs perform near chance on fine-grained binary classification tasks, exposing a fundamental gap in multi-image abstract visual reasoning.

Problem

Existing real-world BP benchmarks either feature coarse-grained concepts (Bongard HOI, Bongard-OpenWorld) that are too easy, or are too small (Bongard-RWR: 60 instances) to support robust evaluation. No scalable benchmark existed that combines realistic imagery with abstract, fine-grained concepts requiring few-shot visual reasoning.

Method

A semi-automated four-stage pipeline generates BP instances from the 60 Bongard-RWR source matrices: 1. Describe: Pixtral-12B generates paired positive/negative captions for each image conditioned on the side's concept \((C_L, C_R)\). 2. Augment: A T2T step diversifies each positive caption into \(N=15\) alternatives that preserve the concept. 3. Generate: Flux.1-dev renders \(512\times512\) candidate images from (positive, negative-prompt) pairs. 4. Review: Human judges discard images that violate concept fidelity.

From the accepted images, \(M=10\) left and \(M=10\) right panel sets are composed per source BP by iteratively selecting subsets that minimize total pairwise cosine similarity of ViT-L/14 embeddings, maximizing intra-set visual diversity. All \(M^2=100\) left×right pairings yield BP instances, giving \(54 \times 100 = 5{,}400\) total matrices (6 source BPs discarded). Six task formulations are defined: Image-to-Side (I1S), Images-to-Sides (I2S), Description-to-Side (D1S), Descriptions-to-Sides (D2S), Concept Selection (CS, multiclass), and Concept Generation (CG, free-form text).

Key Contributions

  • Semi-automated VLM-driven pipeline to generate large-scale BPs with real-world-like imagery and abstract concept fidelity.
  • Bongard-RWR+ dataset: 5,400 BPs across 49 abstract concepts; includes grayscale (Bongard-RWR+/GS) and variable-panel-count (Bongard-RWR+/LP, \(P \in \{2,...,6\}\)) ablation variants.
  • Six BP task formulations covering binary classification, multiclass selection, and free-form generation.
  • Systematic evaluation of InternVL2.5-78B, Qwen2-VL-72B, LLaVA-Next-110B, MiniCPM-o-2.6-8B, and DeepSeek-R1-70B (text-only), with a non-parametric Similarity Classifier (SC) baseline using ViT-L/14 embeddings.

Results

  • Concept Selection (CS): InternVL2.5-78B best at 91% (\(K=2\)), dropping to 57% (\(K=16\)); LLaVA-Next-110B and MiniCPM-o-2.6-8B reach only 19% at \(K=16\).
  • Fine-grained category breakdown: All models retain ~75% on Shape/Size/Branching at \(K=16\); InternVL2.5 drops below 50% for Contour, Rotation, and Angle.
  • Binary classification (I1S/I2S): All VLMs perform at or near chance (0.48–0.50 for I1S); the non-parametric SC baseline (0.52/0.54) outperforms all VLMs on both I1S and I2S.
  • Description-to-Side (D1S): Best result is 0.58 (Qwen2-VL captions + Qwen2-VL prediction); DeepSeek-R1 achieves 0.56–0.57 on text-only D1S/D2S.
  • Model size scaling: Larger models within a family improve CS accuracy but gains plateau; MiniCPM-o-8B matches LLaVA-Next-110B on several settings.
  • Grayscale vs. color: Performance differences negligible (≤+0.06), confirming color is non-essential for abstract concepts originating from synthetic black-and-white BPs.

Limitations

  • Dataset covers only 49 of the original 388 abstract BP concepts; 6 source BPs were discarded due to generation difficulty, introducing selection bias toward visually simpler concepts.
  • Human review ensures concept fidelity but cannot fully guarantee absence of confounding visual cues introduced by Flux.1-dev.
  • Concept Generation (CG) task results are not reported quantitatively in the provided text.
  • Evaluation is restricted to open-access models; closed proprietary VLMs (GPT-4o, Gemini) are not included.
  • Dataset images are AI-generated, not natural photographs, so distribution shift from truly real-world scenes remains.

Relevance to Vision-Language Models

Bongard-RWR+ directly stress-tests VLMs on multi-image few-shot abstract reasoning with fine-grained visual concepts—a regime where all tested frontier models fail at near-chance accuracy despite excelling on standard VLM benchmarks, revealing that scale alone does not confer abstract relational reasoning. The benchmark's six formulations allow fine-grained diagnosis of where VLM pipelines break down (perception vs. language-grounded reasoning), making it a useful diagnostic complement to coarse-grained VLM leaderboards. The finding that a simple ViT-L/14 similarity classifier beats large VLMs on binary classification suggests current VLMs lack the contextual, comparative visual reasoning that humans deploy in few-shot concept induction. This connects to ongoing debates about compositional and relational reasoning in VLMs and motivates work on multi-image attention mechanisms, chain-of-thought visual reasoning, and contrastive multi-image pretraining.