Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges¶

🕒 Published (v1): 2026-07-21 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This survey organizes the emerging field of multimodal visual humor understanding around a three-level capability hierarchy—recognition, interpretation and reasoning, generation—and a two-form representation taxonomy (StaVT and SeqVN). It synthesizes benchmark design, evaluation protocols, and modeling paradigms for MLLMs on memes, cartoons, and comics, then maps the persistent technical and socio-technical barriers. No new model is proposed; the contribution is a principled synthesis that exposes gaps prior surveys leave uncovered.

Problem¶

Existing survey coverage is fragmented: text-only humor surveys address puns and wordplay but ignore visual grounding; task-specific multimodal surveys narrow to sarcasm detection or meme classification alone; broad MLLM surveys subsume figurative understanding as a minor sub-capability without dedicated treatment. The deeper structural gap is that MLLMs are built and evaluated for perceptual alignment, yet multimodal humor requires reasoning about communicative intent—why a juxtaposition is funny, what background knowledge it presupposes, and what stance it communicates—a capability that existing benchmarks and modeling assumptions do not systematically test.

Method¶

The survey proposes three organizing structures applied jointly across the literature:

Representation taxonomy: Static Visual–Textual Artifacts (StaVT), where \(x = \{x_\text{img}, x_\text{text}\}\) (memes, editorial cartoons, captioned images), and Sequential Visual Narratives (SeqVN), where \(x = \{x_1, x_2, \ldots, x_T\}\) (comic strips, multi-panel memes requiring cross-panel entity tracking and causal inference).

Capability hierarchy (three progressively demanding levels): 1. Recognition: binary/multi-class classification, target extraction, intensity estimation; evaluated with accuracy/F1/AUROC. 2. Interpretation and Reasoning: explanation of humor mechanisms (incongruity, analogy, exaggeration, narrative structure); evaluated with BLEU/BERTScore plus rubric-based human or LLM-assisted judgment. 3. Generation: humor-consistent captions, punchlines, or comic continuations conditioned on an input; evaluated primarily by human preference.

Modeling paradigm taxonomy (Table 1): - Cross-modal alignment: \(p_\theta(y \mid x)\) via instruction tuning, contrastive VL pre-training, or MoE routing; scalable but humor mechanism stays implicit. - Grounded reasoning: \(p_\theta(y, r_{1:K} \mid x, K_x)\) via CoT prompting, rationale supervision, theory-guided decomposition (incongruity-resolution model stages), or external knowledge retrieval; transparent but computationally expensive and prompt-sensitive. - Controllable generation: \(p_\theta(y_\text{gen} \mid x, c)\) conditioned on creative control signals (mechanism, tone, target); low controllability in practice, humor quality hard to evaluate automatically.

Four creative meaning mechanisms are defined as background concepts—incongruity, analogy/conceptual mapping, exaggeration/hyperbole, and narrative structure—to anchor which capability each benchmark exercises.

Key Contributions¶

Capability-centric organization that aligns background concepts, benchmark design, and modeling paradigms under one unified framework, distinguishing label prediction from evidence-grounded interpretation.
Synthesis of datasets and evaluation protocols with explicit attention to what current benchmarks do not measure about interpretive understanding (mechanism-level annotation remains sparse).
Identification of four persistent bottlenecks: shortcut-prone evaluation (surface pattern exploitation inflates recognition metrics), weak cultural and narrative coverage, missing evidence grounding in explanations, and unresolved safety/ownership concerns for generated humor.
Formal distinction between StaVT and SeqVN as modeling-oriented representation classes that determine required architectural integration capacity.

Results¶

This is a survey; no new experimental numbers are reported. Cross-benchmark analysis confirms a consistent recognition-to-interpretation drop: models that achieve strong classification accuracy fail to recover the humor mechanism, violated expectation, or social target—demonstrating that discriminative label performance is a weak proxy for genuine understanding. Qualitative findings per paradigm: - Alignment methods: creativity-specific instruction tuning outperforms general-purpose checkpoints on recognition, but performance on nuanced intent lags; text-centric architectures (image → caption → LLM) are surprisingly competitive but lose spatially encoded jokes. - Grounded reasoning: training-time rationale supervision improves both accuracy and interpretability substantially over prompted CoT; prompted CoT is fragile to phrasing and prone to hallucinated rationales. - Generation: reference-based metrics (BLEU, BERTScore) are weak proxies for humor quality; strongest evaluation protocols combine faithfulness to source image, intended target, and rhetorical device.

Limitations¶

Excludes audio- and video-centric humor, generic aesthetic modeling, and persuasive media unless directly used as evaluation resources for image-based humor.
Pre-MLLM task-specific fusion models are only briefly covered in an appendix (Appendix C), not synthesized in the main hierarchy.
The survey is not an exhaustive catalog; coverage follows backward/forward snowballing from benchmark and survey papers, so niche datasets may be missing.
Safety and ownership concerns (e.g., meme copyright, generation of offensive content) are identified as barriers but not resolved.
Evaluation at the interpretation level lacks standard protocols: rubric-based human judgment is expensive, and LLM-as-judge approaches introduce their own reliability concerns.
No quantitative meta-analysis or aggregated cross-benchmark comparison is provided; cross-benchmark findings are qualitative.

Relevance to Vision-Language Models¶

This survey directly targets the frontier where VLMs are structurally weakest—non-literal cross-modal reasoning in culturally grounded artifacts—making it a precise diagnostic map of current VLM limitations beyond standard captioning and VQA benchmarks. The three-paradigm taxonomy (alignment → grounded reasoning → controllable generation) frames the architectural and training-objective choices that determine whether a VLM can move from label prediction to mechanism-aware explanation, which is an open research frontier for next-generation VLMs. The emphasis on evidence grounding, external knowledge retrieval, and theory-guided decomposition (\(p_\theta(y, r_{1:K} \mid x, K_x)\)) connects directly to active VLM research threads on chain-of-thought, tool use, and retrieval-augmented multimodal models. For researchers tracking VLMs, the identified gaps—shortcut-prone benchmarks, sparse mechanism annotations, cultural coverage—provide a concrete agenda for evaluating and stress-testing VLM interpretive capabilities.