Vision-Language Assistant for Emotional Reactions to Risky Driving¶

🕒 Published (v1): 2026-07-17 17:57 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents Keep Yelling Assistant (KYA), a vision–language pipeline that detects risky driving maneuvers (specifically sudden cut-ins) via YOLOv8 and generates emotionally expressive, persona-aligned verbal responses using state-of-the-art LLMs. The system bridges low-level perceptual risk cues with high-level affective interaction by introducing a novel ego-centric risk metric (Projected Reach Time, PRT) and a structured behavior log that conditions LLM response generation. A 108-participant user study evaluates emotional alignment across four LLMs under zero-shot and few-shot prompting.

Problem¶

Existing LLM-based ADAS and VLM-driven autonomous driving systems produce neutral, task-driven outputs that neglect the emotional and psychological dimensions of human driving experience. This limits the realism and effectiveness of human–vehicle interaction: a system that correctly identifies a hazard but fails to convey urgency or reassurance may cause user disengagement and erode trust.

Method¶

KYA operates as a two-stage perception–language pipeline:

Vision module: YOLOv8 variants (n/s/m/l), pretrained on BDD100K and nuScenes and fine-tuned for dashcam deployment (confidence threshold lowered to 0.3 for small/occluded vehicles), detect and track all visible vehicles at 5–10 fps. For each frame, the system identifies the "aggressor" vehicle (highest combination of proximity and closing speed) and extracts spatial/contextual attributes. A novel ego-centric risk metric, Projected Reach Time, is computed as:

\[\text{PRT} = \frac{d_{\text{rel}}}{v_{\text{ego}} + \varepsilon}\]

where \(d_{\text{rel}}\) is the relative distance (m) and \(v_{\text{ego}}\) is ego-vehicle speed (kph). Unlike TTC or PET, PRT requires only the ego vehicle's own trajectory, making it suitable for single-camera dashcam setups. All features are compiled into a structured behavior log.

Language module: The behavior log plus a user-selected emotional persona (neutral, mildly annoyed, moderately annoyed, highly angry, humorous, magnanimous, analytical) are passed to one of four LLMs (ChatGPT-4o, Claude 3, Gemini 2.5, Copilot). The LLM performs two-stage reasoning: situation interpretation (severity, cut-in direction, PRT trend) followed by persona-aligned natural language reaction generation. Output is passed to a TTS engine with vocal characteristics matched to the selected persona.

Key Contributions¶

A full perception–behavior–language architecture (KYA) that couples real-time dashcam-based risk detection with emotionally expressive LLM response generation.
A practical YOLO-based cut-in detection module operable on unconstrained dashcam footage with scale-invariant normalization.
The Projected Reach Time (PRT) metric: an ego-centric, single-camera-compatible temporal proximity indicator sensitive to abrupt cut-in events.
A structured behavior log interface that decouples the vision and language modules, enabling modular LLM substitution.
A 108-participant user study evaluating persona preference and LLM emotional alignment across seven emotional tones and four LLMs under zero-shot and few-shot conditions.

Results¶

YOLOv8l achieved best detection: recall 0.869, F1 0.921, mAP@0.5 0.998, MOTA 0.854, avg. IoU 0.988, only 1 ID switch.
YOLOv8s offered the best real-time deployment trade-off: precision 0.982, recall 0.762, F1 0.858, mAP@0.5 0.991, MOTA 0.745, avg. IoU 0.969, 1 ID switch.
YOLOv8n was weakest: recall 0.737, MOTA 0.715.
The combination of YOLOv8s + ChatGPT-4o achieved the highest overall user rating of 4.29/5.00 on the 5-point Likert scale (fluency, emotional alignment, persona consistency).
All four LLMs received favorable ratings, though persona preference distributions varied across participant groups.
Dataset: 20 dashcam video sequences, 5,019 total frames, 1,189 labeled high-risk frames.

Limitations¶

Dataset is small (20 video clips, all cut-in scenarios only) and sourced from YouTube, limiting generalization to other risky maneuver types and geographic contexts.
PRT is not a physically calibrated collision metric; it depends on normalized, scale-invariant visual estimates rather than precise depth or velocity sensors, reducing absolute accuracy.
The behavior recognition module assumes at least one cut-in event per scene and identifies only the single highest-risk aggressor per frame, not handling simultaneous multi-vehicle threats.
User study relies on self-reported persona preferences rather than measuring actual safety outcomes or real-world emotional regulation effectiveness.
The paper is truncated and does not report complete few-shot vs. zero-shot comparison results or per-persona LLM breakdowns.
No ablation of the TTS component or assessment of how vocal tone affects perceived emotional alignment.

Relevance to Vision-Language Models¶

KYA demonstrates a practical modality-bridging application of VLMs: structured visual perception outputs (bounding boxes, PRT, behavior logs) serve as grounded, language-readable prompts that condition LLM affective reasoning without requiring end-to-end vision-language training. This is relevant to the VLM community as it illustrates a pipeline architecture where lightweight detection models feed interpretable intermediate representations to general-purpose LLMs, bypassing the need for a unified VLM backbone while still achieving contextually grounded language generation. The comparative evaluation of ChatGPT-4o, Claude 3, Gemini 2.5, and Copilot on the same structured-input task provides a rare applied benchmark of frontier LLMs on affective, safety-critical automotive dialogue. For researchers tracking VLMs in autonomous driving, KYA occupies a niche currently underserved by scene-understanding benchmarks (DriveLM, NuScenes-QA): emotionally adaptive human–vehicle interaction rather than pure perception or planning.