Skip to content

HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HoloLLM is a Multimodal Large Language Model that integrates rare sensing modalities—LiDAR, infrared, mmWave radar, and WiFi-CSI—for language-grounded human action recognition, QA, and captioning in smart-home settings. The core challenge is aligning data-scarce, heterogeneous sensor signals with text without large-scale pretraining. The proposed Universal Modality-Injection Projector (UMIP) overcomes this via coarse-to-fine cross-attention between CLIP-prealigned embeddings and modality-specific convolutional features.

Problem

Standard VLMs rely on visual data, making them brittle under occlusions, low light, and privacy constraints. Extending MLLMs to non-visual sensing modalities (mmWave, WiFi, LiDAR, infrared) is blocked by two obstacles: (1) only thousands of lab-collected samples exist for these modalities vs. millions for RGB; (2) their signal representations are physically heterogeneous, making transformer-based universal encoders insufficient for discriminative feature extraction.

Method

HoloLLM processes each sensing input through two parallel branches: (a) a frozen CLIP ViT-L encoder that produces pre-text-aligned initial embeddings (exploiting CLIP's transferability to out-of-domain modalities), and (b) modality-specific convolutional tailored encoders (ResNet18 for video/depth/infrared, PointNet for LiDAR/mmWave, 1D Temporal ResNet18 for RFID, MetaFi for WiFi) pretrained on HAR classification. The UMIP projector takes average-pooled CLIP embeddings as coarse queries and the tailored encoder feature maps as keys/values, running L=8 blocks of self-attention → cross-attention → FFN to iteratively inject fine-grained modality-specific features into the text-aligned query stream. The final tokens are projected via MLP (dim 1024→4096) into LLaMA2-7B's embedding space. Training is two-stage: (1) supervised pretraining of tailored encoders with cross-entropy HAR loss; (2) fine-tuning UMIP + tokenizers with combined HAR classification + next-token prediction loss. Text annotations for sensing datasets (MM-Fi, XRF55) are generated via a human-VLM collaborative pipeline using LLaVA-Video with in-context learning for captions and GPT-4o-rewritten question templates for QA.

Key Contributions

  • HoloLLM: first MLLM to unify rare sensing modalities (LiDAR, mmWave, WiFi-CSI, infrared, RFID) with language-grounded human perception and reasoning.
  • Universal Modality-Injection Projector (UMIP): coarse-to-fine cross-attention design that leverages CLIP's text-alignment as initialization and injects discriminative features from tailored encoders, requiring only minimal task-specific fine-tuning data.
  • Human-VLM collaborative data curation pipeline for generating action QA and caption annotations on MM-Fi and XRF55 sensing datasets.
  • First multisensory benchmark for language-grounded human sensing, covering three tasks (action recognition, action QA, action captioning) under random, cross-subject, and cross-environment splits.

Results

  • On MM-Fi Action QA (CrossEnv, average across 5 modalities): HoloLLM 56.4% vs. ImageBind 16.7%, OneLLM 5.0%, Honeybee 1.7%, Tokenpacker 4.6%.
  • On XRF55 Action QA (Random, average): HoloLLM 63.5% vs. ImageBind 35.8%, OneLLM 1.8%, others ~1.2–1.5%.
  • On MM-Fi Action Caption METEOR (CrossEnv, average): HoloLLM 22.6% vs. ImageBind 17.3%, OneLLM 9.3%, Tokenpacker 3.8%.
  • HoloLLM improves existing MLLMs by up to ~30% on some QA tasks (paper-stated headline number).
  • Ablation (MM-Fi CrossEnv): baseline Q-Former avg Action QA 6.2% → +TailoredEncoder 46.6% → +UMIP 56.4%; UMIP contributes especially on QA (+9.8 pp avg).
  • tSNE visualization confirms UMIP produces better-separated action clusters and closer alignment between sensing and text tokens vs. baselines.
  • Performance degrades on WiFi and RFID under CrossSub/CrossEnv, indicating environment/subject sensitivity is an unsolved challenge for these modalities.

Limitations

  • Tasks are restricted to action recognition, QA, and captioning; higher-level capabilities (task planning, agent action generation) are not addressed.
  • WiFi-CSI and RFID modalities show limited cross-subject and cross-environment generalization due to signal sensitivity to physical environments and human body variability.
  • Benchmarks are built on only two datasets (MM-Fi, XRF55) with lab-collected sensing data; real-world deployment scale is untested.
  • Text annotations are auto-generated (LLaVA-Video + GPT-4o), introducing potential noise in caption quality that is not separately quantified.
  • LLaMA2-7B backbone is fixed; scalability to larger or newer LLMs is not evaluated.

Relevance to Vision-Language Models

HoloLLM directly extends the VLM paradigm to non-visual modalities by exploiting CLIP's cross-modal transferability as a bootstrap mechanism for text alignment under data scarcity—a principled alternative to large-scale pretraining that is relevant to any VLM work dealing with low-resource or out-of-domain sensor inputs. The UMIP architecture offers a generalizable projector design that decouples text alignment (via pre-trained CLIP) from discriminative feature extraction (via task-specific encoders), potentially applicable wherever VLMs must ingest heterogeneous physical signals. For researchers tracking VLMs, this work highlights a concrete pathway toward embodied, multisensory agents that retain natural language reasoning while overcoming visual limitations such as occlusion and lighting—directly relevant to embodied AI and robot perception threads in the VLM literature.