Skip to content

Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Serge Belongie (Cornell Tech); ICLR 2026; comprehensive fine-grained LVLM evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FG-BMK is a large-scale fine-grained evaluation benchmark for LVLMs comprising 1.01 million questions and 0.28 million images sourced from 12 established fine-grained datasets. It evaluates models from both human-oriented (conversational Q&A on visual attributes and category hierarchies) and machine-oriented (retrieval and classification) perspectives. Extensive experiments across twelve LVLMs/VLMs reveal systematic weaknesses in fine-grained discriminability that existing holistic benchmarks obscure.

Problem

Existing LVLM benchmarks (MMBench, LVLM-eHub, GQA, DocVQA) assess general multimodal perception and reasoning but do not systematically probe fine-grained visual recognition—tasks that require distinguishing subordinate-category-level visual differences. As a result, the capability boundaries of LVLMs on fine-grained tasks remain poorly characterized, and failure modes such as knowledge bias, attribute blindness, and alignment-induced discriminability loss are not quantified.

Method

FG-BMK aggregates images from 12 curated fine-grained datasets and structures evaluation along two axes:

Human-oriented evaluation uses three question types (true/false, multiple-choice, short-answer) across three subtasks: - Attribute Recognition: tests identification of visual attributes (color, pattern, shape, size, length) using T/F and MC questions derived from dataset annotations. - Knowledge Bias Estimation: exclusively T/F questions pairing each fine-grained category label against same-superclass distractors to surface per-category accuracy inconsistencies. - Hierarchical Granularity Recognition: questions at class, genus, and species levels with distractors drawn from the same parent node in the taxonomy.

Machine-oriented evaluation uses the DINOv2 protocol (mAP for retrieval, Top-1 accuracy for classification) to directly measure visual feature discriminability. It includes (a) within-meta-category retrieval/classification and (b) cross-meta-category classification where subcategories from different superclasses are merged into a unified set. Robustness is probed by applying projected gradient descent (PGD) perturbations to visual features and measuring accuracy degradation. Vision-language alignment effects are isolated by comparing original vs. text-aligned visual features on classification tasks.

Key Contributions

  • FG-BMK benchmark: 1.01M questions, 0.28M images, covering attribute recognition, knowledge bias, hierarchical granularity recognition, image retrieval, and image classification.
  • Dual-paradigm evaluation (human-oriented + machine-oriented) isolating semantic reasoning from feature representation quality.
  • Systematic empirical analysis of 12 LVLMs/VLMs revealing the effects of training paradigm (contrastive vs. generative vs. reconstruction), modality alignment, encoder scale, and PGD perturbation on fine-grained task performance.
  • Diagnosis that knowledge bias in LVLMs stems from uneven fine-grained category representation in training data, validated by fine-tuning on occurrence-balanced datasets.

Results

  • Hierarchical granularity degradation (InternVL3 on CUB-200-2011 MC): class-level 99.76% → genus-level 90.75% → species-level 61.18%.
  • Knowledge bias: LLaVA accuracy per fine-grained bird category spans ~30%–90%; fine-tuning on balanced data eliminates this spread.
  • Attribute recognition (InternVL3, CUB-200-2011): pattern avg. 50.13%, size 52.55%, length 71.03%, shape only 30.95%; some shape attributes score as low as 10.42%.
  • LVLMs vs. fine-grained tailored models (linear classifier / short-answer vs. SOTA):
  • CUB-200-2011: LC 91.65% / SA 85.60% vs. tailored 93.10%
  • Stanford Dogs: LC 90.50% / SA 86.49% vs. tailored 97.30%
  • Stanford Cars: LC 94.30% / SA 90.55% vs. tailored 97.10%
  • Food-101: LC 95.67% / SA 95.25% vs. tailored 98.60%
  • FGVC Aircraft: LC 78.88% / SA 66.19% vs. tailored 95.40%
  • Training paradigm: contrastive-trained models (EVA-CLIP, CoCa) outperform generative/reconstruction-trained models on retrieval and classification across the 12 datasets (Figures 4–5, Nemenyi test groupings in Figures 6–7).
  • Alignment effect: text-aligning visual features degrades fine-grained classification accuracy, especially when image-text training pairs lack fine-grained granularity (Table 4, text truncated before values).
  • Robustness: LVLMs exhibit greater accuracy drop under PGD perturbations on fine-grained tasks than on generic vision tasks (Table 5, text truncated before values).

Limitations

  • Benchmark questions are generated from existing dataset annotations via templates; human-verified question quality at scale is not reported.
  • Human-oriented short-answer scoring uses a substring-match heuristic ("correct if response includes ground truth"), which may over-credit partial or coincidental matches.
  • The paper evaluates only 12 models; closed-source models (GPT-4o, Gemini-2.0-flash) are included only in human-oriented tasks and not in machine-oriented feature-level analysis.
  • Cross-meta-category classification conflates domain shift with discriminability, making it difficult to isolate the contribution of each factor.
  • Fine-grained datasets used are predominantly Western (birds, cars, aircraft); coverage of other cultural or biological domains is limited.
  • The paper text is truncated before presenting full machine-oriented numerical results (Tables 4 and 5).

Relevance to Vision-Language Models

FG-BMK fills a critical blind spot for VLM researchers: existing holistic benchmarks do not stress-test subordinate-level visual discrimination, so models can score well overall while failing at species- or part-level granularity. The finding that text-alignment during training degrades fine-grained feature discriminability has direct implications for how VLMs are pre-trained and what image-text pairing strategies are used. The demonstration that contrastive objectives preserve discriminability better than generative ones informs architectural and training recipe choices for the next generation of VLMs. Results also establish concrete gap targets (e.g., ~17% on FGVC Aircraft between current LVLM linear probing and tailored SOTA) that benchmark progress in fine-grained VLM capability.