Skip to content

VaseMuseum: Digital Intelligent Museum for Ancient Greek Pottery

🕒 Published (v1): 2026-07-07 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VaseMuseum is a training-free multimodal agent framework that wraps a frozen VLM with a multi-stage inference-time harness — iterative retrieval, deterministic source filtering, response auditing, and GRPO-style multi-rollout selection — to reduce hallucination and improve citation validity in open-domain digital museum interaction with ancient Greek pottery. It treats external knowledge as evidence to be verified rather than text to be copied, triggering evidence-bounded hedging when retrieved support is insufficient or conflicting.

Problem

Open-ended cultural-heritage QA exposes two gaps in standard VLMs: (1) retrieval-augmented generation imports low-quality, inaccessible, or unverifiable sources into the context without any filtering; (2) VLMs produce fluent but overconfident answers when domain evidence is incomplete, disputed, or underspecified — a setting where expert caution is required. Existing closed-benchmark approaches (VaseVQA, VaseVQA-3D) do not transfer to free-form museum conversation with variable evidence quality.

Method

VaseAgent is an inference-time harness layered around a frozen VLM backbone with four coordinated stages:

  1. DeepResearch-style iterative retrieval loop: Given query \(q\) and artifact observation \(I\), the VLM decomposes the request into targeted search intents, issues web/museum-source queries, and refines over up to \(R_{\max}\) rounds conditioned on intermediate reasoning — not a single-shot RAG call.

  2. Source control: Each retrieved hit \(h\) is scored by $\(s_{\text{val}}(h) = 0.55\,a(h) + 0.45\,c(h),\quad h \in H_{\text{val}} \iff s_{\text{val}}(h)\geq 0.40 \text{ and } a(h)\geq 0.50,\)$ where \(a(h)\) is a URL accessibility proxy and \(c(h)\) is text sufficiency. Survivors are ranked by a pre-diversity score \(\phi_{\text{pre}}(h)\) combining query relevance, source quality, and accessibility, then up to \(N_{\text{src}}=5\) hits are selected by Maximal Marginal Relevance (MMR) with \(\lambda_{\text{MMR}}=0.72\).

  3. Response control: The draft answer is decomposed into claim units; lexical overlap \((\tau_{\text{sup}}=0.08)\) determines per-claim evidence support. A composite confidence score $\(\psi = 0.40\,c_{\text{cov}} + 0.30\,\kappa_{\text{cons}} + 0.20\,\omega_{\text{multi}} + 0.10\,m_E - \min(0.35,\,0.12\,n_{\text{conf}})\)$ triggers "uncertain mode" with an evidence-bounded preamble when \(\psi < \tau_{\text{conf}}=0.52\).

  4. Training-free GRPO-style selection: When inference budget allows, \(K=4\) independent controlled rollouts are sampled; each candidate is scored by lightweight verifiers (link validity, claim-evidence support, neutrality, conflict awareness), and the best is selected via \(k^* = \arg\max_k R(a_k, \tau_k, q, I)\) — no gradient updates to the VLM.

Key Contributions

  • VaseMuseum end-to-end framework connecting a virtual 2D/3D museum interface with an open-domain multimodal reasoning agent
  • VaseAgent: inference-time agent combining iterative DeepResearch-style retrieval with deterministic source-level filtering (validity + MMR diversity)
  • Response-level reliability control: composite confidence score \(\psi\) with explicit conflict-penalty term drives evidence-bounded hedging
  • Training-free GRPO-style multi-rollout selector: reranks frozen-model outputs by structured reliability verifiers without fine-tuning
  • Evaluation dataset: 518 LIMC-linked vases from VaseVQA-3D, balanced at 100 instances per task category (visual-only, visual+knowledge, ambiguous)

Results

The paper text is truncated before the quantitative results table; specific numbers are unavailable. Qualitatively reported against search-enabled VLM baselines: - Improved link validity (fraction of cited URLs that are accessible and relevant) - Reduced hallucination rate on visual+knowledge queries - More neutral/calibrated responses on ambiguous questions - GRPO-style selection (\(K=4\)) provides additional reliability gain over the single-trajectory baseline - Qualitative examples show the agent avoids fabricated or over-specific explanations by grounding claims in retrieved evidence

Limitations

  • Evaluation set is small (100 instances per category, 300 total) and restricted to ancient Greek pottery — generalization unverified
  • Source-control and response-control thresholds (\(\tau_{\text{sup}}\), \(\tau_{\text{conf}}\), \(\tau_{\text{keep}}\), \(\lambda_{\text{MMR}}\)) are manually set, not learned
  • GRPO-style selection multiplies inference cost by \(K=4\times\); must be disabled under latency constraints
  • Quantitative baseline comparisons are not available in the provided text (truncated)
  • Domain-expert annotation is limited to spot-checking; full inter-annotator agreement not reported

Relevance to Harnesses / Meta-Harnesses

VaseMuseum is a concrete instance of a domain-specific inference-time meta-harness: a fixed orchestration pipeline (retrieve → filter → generate → audit → select) that wraps a frozen backbone to deliver reliability guarantees the base model cannot provide alone — the defining harness pattern. The GRPO-style multi-rollout selector — sample \(K\) independent trajectories, score each with structured verifiers, pick best — is a direct instantiation of "best-of-N with composite reward" as a harness-layer primitive, entirely decoupled from model training. The modular architecture (each stage independently replaceable) and the per-stage audit log for failure attribution are design principles transferable to general meta-harness engineering, particularly for knowledge-intensive pipelines where retrieval quality and uncertainty calibration must be controlled independently.