A VLM-Enhanced Framework for Comprehensive Traffic Sign Condition Assessment Integrating Daytime Visual Performance and Nighttime Retroreflectivity Evaluation¶

🕒 Published (v1): 2026-07-07 16:33 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a unified pipeline for traffic sign condition assessment that fine-tunes three VLMs (Qwen2.5-VL, InternVL, LLaVA-NeXT-Video) for daytime visual scoring across four MUTCD-defined factors, then fuses those scores with LiDAR-derived retroreflectivity estimates into a single Sign Condition Index (SCI). The hybrid scoring converts free-text VLM outputs into numbers via sentiment analysis and CLIP similarity. Applied to 462 real-world signs, the framework flagged 68 for immediate replacement on retroreflectivity grounds while avoiding costly handheld retroreflectometers.

Problem¶

No prior automated system jointly evaluates both daytime visual performance (legibility, color fidelity, surface/shape integrity, environmental occlusion) and nighttime retroreflectivity in a single framework. Existing methods treat these as separate problems; image-based pipelines ignore retroreflectivity, and LiDAR-based retroreflectivity tools use imagery only for color segmentation rather than visual condition analysis. Manual inspections are subjective and costly; handheld retroreflectometers are unaffordable for small municipal agencies.

Method¶

Data. Two-phase collection in Columbia, MO: (1) sign images for VLM fine-tuning (647 undamaged + 89 manually damaged + 317 from a public dataset + 300 augmented = 1,353 total; 70/30 train/test split); (2) synchronized GoPro 11 + Livox HAP LiDAR acquisition for fusion, aligned via ROS timestamps and 32 spatial correspondence points producing an extrinsic transformation matrix.

Daytime VLM assessment. Three VLMs are fine-tuned with LoRA adapters (frozen base weights) using prompt-based supervision and region-specific sign crops. Each model generates five descriptive terms per factor across four factors (legibility, color, surface/shape integrity, surrounding environment). The fine-tuning targets four severity levels per factor (slight/moderate/severe × augmentation type). Model outputs are converted to numerical scores via (a) sentiment analysis on the generated text and (b) CLIP cosine similarity between the predicted description and reference condition descriptions.

Nighttime retroreflectivity. LiDAR retro-intensity values are extracted from fused point clouds associated with each sign region, then converted to estimated retroreflectivity (cd/lux/m²) following the calibration procedure of Ai & Tsai.

Integration. Per-factor daytime scores and the retroreflectivity estimate are combined into a Sign Condition Index (SCI) that outputs maintenance recommendations, including a binary flag for signs below MUTCD minimum retroreflectivity thresholds.

Key Contributions¶

First framework to integrate MUTCD-defined daytime visual factors with LiDAR-derived nighttime retroreflectivity into a unified SCI
Fine-tuning methodology for three open-source VLMs (Qwen2.5-VL, InternVL, LLaVA-NeXT-Video) on a balanced, augmented traffic sign dataset with cross-factor damage annotations
Hybrid scoring pipeline translating open-ended VLM text into standardized numerical condition scores via sentiment analysis and CLIP scoring
Cost-effective alternative to retroreflectometers using commodity Livox HAP LiDAR plus camera

Results¶

LLaVA-NeXT-Video and Qwen2.5-VL achieved bidirectional cosine similarity scores of 0.67–0.76 across all four daytime assessment factors; InternVL performed below both
Among 462 validated traffic signs, 68 (14.7%) were flagged as requiring immediate replacement due to retroreflectivity below MUTCD thresholds
(Full per-factor breakdown and comparison tables referenced but not reproduced in the provided excerpt)

Limitations¶

Paper text is truncated; stated limitations section not available in the provided excerpt
Augmented damage images are synthetic proxies; real-world distribution of damage types and severities may differ
LiDAR retroreflectivity estimation relies on Ai & Tsai calibration, which was shown to be sensitive to environmental conditions in prior work
Dataset collected under clear weather in a single city (Columbia, MO), limiting geographic and weather generalizability
VLM evaluation metric (cosine similarity 0.67–0.76) reflects semantic alignment with reference descriptions, not direct comparison to ground-truth human inspector ratings
Retroreflectivity thresholds applied are MUTCD minimums; sign-type-specific thresholds (which vary by sign class) are not discussed in detail

Relevance to Vision-Language Models¶

This work is a concrete applied case of fine-tuning open-source VLMs (Qwen2.5-VL, InternVL, LLaVA-NeXT-Video) with LoRA for a structured visual inspection task, demonstrating that domain-specific prompt supervision and region-cropped inputs can steer general VLMs toward categorical, multi-attribute assessment. The hybrid scoring approach—converting free-form VLM outputs into numerical scores via CLIP similarity—is a transferable design pattern relevant to any VLM deployment requiring scalar outputs from language generation. It also benchmarks three prominent open-weight VLMs on a real-world perception task, offering a domain-specific data point for model selection. For VLM researchers, the bidirectional cosine similarity evaluation protocol and the dataset augmentation strategy for condition severity may inform work on robustness and out-of-distribution generalization.