Multi-modal Rail Crossing Safety Analysis¶

🕒 Published (v1): 2026-07-01 18:26 UTC · Source: Arxiv · link

Why this paper was selected

Multimodal safety analysis fusing vision and structured accident reports; applied safety-critical domain

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a proof-of-concept multimodal pipeline for highway–rail grade crossing safety assessment, combining street-level imagery with FRA Form 57 accident records and VLMs. A LoRA fine-tuned Gemma 4 E4B model is used to predict FRA-style APS risk scores and classify crossings as HIGH-RISK or LOW-RISK. The best system achieves a macro F1 of 0.757 for binary classification and RMSE of 0.078 with Pearson correlation of 0.492 for continuous score regression.

Problem¶

Existing railway crossing risk models (e.g., APS/GXAPS) rely on predefined structured variables (train counts, AADT, speed) that omit visual and context-dependent cues observable to highway users—such as sightlines, warning-device visibility, road geometry, and environmental conditions. Manual expert inspection does not scale to the tens of thousands of U.S. grade crossings. Prior VLM safety work focuses on train-cab or road contexts, not the highway-user perspective at grade crossings.

Method¶

Data: 1,634 Mapillary street-level panoramic images from 149 California crossings, paired with 406 FRA Form 57 accident/incident records (1975–2023). Ground-truth risk scores come from FRA's GXAPS (APS-based annual average predicted accidents).

Score-based analysis: - Base model: Gemma 4 E4B, fine-tuned via LoRA on image sequences + Form 57 records. - Binary classifier predicts LOW-RISK vs. HIGH-RISK (threshold: APS score > 0.09). - Continuous regressor predicts the annual average predicted accident score. - Routed regression: the binary classifier routes each crossing to a specialized LOW-RISK or HIGH-RISK regressor, each independently LoRA fine-tuned on its risk stratum. - Baselines: image-only prompting, image+Form 57 prompting, oracle prompting (images + Form 57 + latent variables AADT/train count/speed).

Description-based analysis: - Bowtie-diagram-grounded structured prompting: VLM identifies threats, safety barriers (warning devices, signage), and escalation factors (sun glare, vegetation, shadow) from the highway-user viewpoint. - Images are augmented via Gemini to simulate rain, strong sunlight, and nighttime, testing how environmental variation affects risk-factor visibility.

Key Contributions¶

A multimodal pipeline combining street-level crossing imagery with FRA Form 57 records for scalable crossing safety assessment.
LoRA fine-tuning of Gemma 4 E4B for VLM-based APS risk score estimation from multimodal crossing data.
A routed regression architecture that first classifies risk tier, then applies specialized regressors per tier to handle the long-tailed score distribution.
Accident-theory-grounded (bowtie diagram) structured visual inspection protocol for identifying threats, barriers, and escalation factors.
Evaluation under image augmentation (rain, sunlight, night) to assess environmental robustness.

Results¶

Binary classification (Table 1): - E4B image-only: macro F1 = 0.494, HIGH-RISK F1 = 0.176 - E4B + Form 57: macro F1 = 0.523, HIGH-RISK recall improves to 0.460 - E4B oracle (+ AADT/trains/speed): HIGH-RISK recall = 0.921, but precision remains low (0.270); macro F1 = 0.516 - E4B fine-tuned: accuracy = 0.851, balanced accuracy = 0.767, HIGH-RISK F1 = 0.607, macro F1 = 0.757 (best)

Continuous score regression (Table 2): - E4B image-only: RMSE = 0.562, Pearson \(r\) = −0.011, \(R^2\) = −95.9 - E4B + Form 57: RMSE = 0.432, \(r\) = 0.014 - E4B oracle: RMSE = 0.254, \(r\) = 0.052 - E4B fine-tuned: RMSE = 0.071, \(r\) = 0.245 - Routed regressor: RMSE = 0.078, \(r\) = 0.492, \(R^2\) = 0.067 (best deployable) - Oracle routed regressor (upper bound): RMSE = 0.060, \(r\) = 0.756, \(R^2\) = 0.536

Limitations¶

Dataset is small (149 crossings, California only), limiting generalizability.
Prompting-only baselines yield near-zero or negative correlation with APS scores, suggesting off-the-shelf VLMs cannot perform this task without fine-tuning.
The routed regressor's \(R^2\) (0.067) is barely positive; the large gap to the oracle router (\(R^2\) = 0.536) indicates routing errors are a major bottleneck.
Several APS input variables (AADT, train count, speed) are not inferable from imagery or Form 57, creating an information ceiling for purely visual approaches.
Image data is crowd-sourced (Mapillary), introducing variable viewpoint, focal length, and geolocation noise.
The paper is a proof-of-concept; the description-based analysis results are reported qualitatively only.

Relevance to Vision-Language Models¶

This work provides a concrete domain application of fine-tuned compact VLMs (Gemma 4 E4B with LoRA) for structured multimodal regression, demonstrating that task-specific adaptation is necessary—zero-shot and few-shot prompting fail badly on a quantitative prediction task even with relevant context injected. The routed prediction strategy offers a practical template for handling skewed label distributions in VLM regression pipelines. For VLM researchers, it highlights the gap between qualitative scene-understanding capability (which VLMs possess) and precise numerical estimation (which requires fine-tuning and task-specific architectural choices). The bowtie-diagram-grounded inspection protocol also illustrates how domain knowledge can be encoded into structured VLM prompts for safety-critical applications.