Skip to content

Teaching Human Behavior Improves Content Understanding Abilities Of VLMs

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper shows that fine-tuning VLMs to predict receiver behaviors (likes, comments, replay graphs) collected from Reddit and YouTube at scale improves their content-understanding across a broad range of downstream tasks. The key insight is that user engagement signals are downstream effects of content and thus encode rich semantic information about that content. The resulting model, Behavior-LLaVA, outperforms its LLaMA-Vid base on 46 tasks across 26 benchmarks without any architectural changes.

Problem

Prior VLM training pipelines discard behavioral metadata (comments, likes, view counts) as noise, even though such signals are downstream effects of content and therefore carry latent semantic information. Perceptual signals (eye-tracking, fMRI) that have been studied for this purpose are lab-constrained and too small to scale to LLM training regimes. The gap is that large-scale, freely available action-level behavioral signals from the internet remain unexploited for improving content understanding.

Method

The authors collect the BLIFT dataset (730k image/video samples) from Reddit (r/pics, r/videos) and YouTube (curated via Wikidata), pairing media with receiver behaviors: upvotes, likes, view counts, top comments, and replay graphs. Extensive filtering removes NSFW, bot-generated, duplicate, and low-engagement content. They formulate an instruction fine-tuning task: given a video/image (plus ASR if available), the model must generate scene-by-scene descriptions and then simulate predicted like ratios and top-5 user comments. LLaMA-Vid is fine-tuned on BLIFT mixed with its original SFT data at a 1:1 ratio for 2.2 epochs, with all modules except the visual encoder frozen. A control model (Ad-LLaVA) is trained on the same media without any behavioral labels to disentangle the effect of additional data from the effect of behavioral supervision.

Key Contributions

  • BLIFT dataset: 400k images + 330k videos from Reddit and YouTube paired with receiver behavior (comments, likes, replay graphs), released publicly.
  • Behavior-LLaVA: LLaMA-Vid fine-tuned on behavior prediction; no architectural changes required.
  • Controlled ablation (Ad-LLaVA): Demonstrates that behavioral labels—not merely additional training data—drive performance gains.
  • Perception vs. action ablation: Shows action-level behavior (comments, likes) substantially outperforms perception-level behavior (Salicon saliency, ~10k images) due to scale and diversity advantages.
  • Breadth of evaluation: 46 tasks, 26 benchmarks spanning image, video, audio, and text modalities, in both zero-shot and fine-tuned settings.

Results

  • Long Video Understanding (LVU): Behavior-LLaVA improves over LLaMA-Vid on 9/9 tasks, average +21.49%; outperforms state-of-the-art on 5/9 tasks.
  • Video emotion (VideoEmotion-8, CAER, Ekman-6): +51.85% average improvement zero-shot over LLaMA-Vid; outperforms supervised baselines on 3/3 zero-shot, 1/3 fine-tuned.
  • Memorability (7 benchmarks—LaMem, SUN, MemCat, Memento10k, VideoMem, MediaEval, LAMBDA): +186.4% zero-shot improvement over LLaMA-Vid; matches SOTA with only 25% of training data in fine-tuned setting.
  • Audio/text generalization: +19.5% on audio summarization + IMDB sentiment (Table 12).
  • Dense captioning: Behavior-LLaVA approaches LLaVA-34B (2.5× larger) on detail and quality metrics.
  • Ad-LLaVA performs on par with the LLaMA-Vid base, confirming behavioral labels are the source of gains.
  • Comment perplexity drops from 6.22 → 3.05; like/view R² improves from −5.1 → 0.45 after fine-tuning.

Limitations

  • Evaluation is anchored to one base architecture (LLaMA-Vid); generalizability to other VLM families is not demonstrated.
  • Data is limited to English-language content from Reddit and YouTube, introducing platform and cultural biases.
  • ~51% of originally collected YouTube videos were removed or made private by collection time, reducing diversity.
  • Behavior-LLaVA shows a decrease in captioning correctness (factuality/hallucination metric) relative to LLaMA-Vid despite gains in detail and quality.
  • Perception-level behavior (saliency) does not consistently improve results, and the scale argument relies on the gap between 10k lab samples vs. internet data—not a principled analysis of signal content.
  • The instruction template auto-generates scene descriptions using LLaVA-13B, introducing noise from a weaker teacher model.

Relevance to Vision-Language Models

This paper directly addresses a training data design choice for VLMs: whether to include or discard user engagement metadata. It demonstrates that behavioral supervision is a scalable, annotation-free strategy for improving high-level semantic understanding—emotion, persuasion, memorability—which are precisely the tasks where VLMs still lag behind specialized supervised models. The work connects to instruction fine-tuning research (LLaVA, MiniGPT-4) by showing that the type of supervision signal matters beyond data volume. For researchers tracking VLMs, the BLIFT dataset and the perception-vs-action ablation together provide a useful framework for thinking about what implicit signals in web-scale data can substitute for expensive human annotation.