Skip to content

Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector

🕒 Published (v1): 2026-07-08 13:15 UTC · Source: Arxiv · link

Why this paper was selected

Zero-shot VLM benchmark vs. specialized FRB detector; honest generalist vs. specialist evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks small, open-weight, locally-run Vision-Language Models (Gemma 4 2B and 4B) against the specialized detector SwinYNet for Fast Radio Burst (FRB) detection in dynamic spectra under a strict zero-shot, prompt-only regime. On a balanced binary benchmark of 2000 simulated L-band spectra, Gemma 4 2B achieves 93.65% accuracy—statistically indistinguishable from SwinYNet's 92.90%—while exhibiting a markedly lower false-positive rate on structured RFI. The result establishes a strong lower bound on the zero-shot visual transfer capability of generalist VLMs to specialized astrophysical signal classification.

Problem

Specialized deep learning FRB detectors (e.g., SwinYNet) achieve exceptional performance but require large task-specific training datasets and cannot be repurposed without full retraining. Traditional dedispersion pipelines scale as \(\mathcal{O}(N_t N_\nu N_{DM})\) and produce high false-positive rates from RFI. The open question is whether a generalist VLM—with no domain-specific training—can already recognize the characteristic \(\nu^{-2}\) dispersive sweep of an FRB in a dynamic spectrum, removing the need for labeled astronomical datasets.

Method

Two open-weight, locally-run VLMs (Gemma 4 2B and 4B) are applied in a zero-shot, prompt-only regime: model weights are frozen from general multimodal pretraining; no fine-tuning, few-shot examples, RAG, or tool calls are used. The model receives a single dynamic-spectrum PNG image and a single textual instruction and returns a structured JSON response containing a binary label, a continuous FRB probability, a confidence score, and a natural-language justification of visual attributes.

The benchmark consists of 3000 simulated L-band PSRFITS files generated with simulateSearch, covering FRB signals (with recorded DM, arrival time, width, flux, profile type), structured RFI (narrowband, broadband, periodic, multiband), and pure noise. A balanced binary subset of 2000 samples (FRB vs. NON-FRB) is used for the paired VLM–SwinYNet comparison; the full 3000-sample set is used for three-class (FRB/RFI/noise) evaluation. Each simulated file covers 2 s of observation with 2048 frequency channels, time sampling \(\Delta t = 1.96608 \times 10^{-4}\) s, and 8-bit quantization.

VLM performance is compared sample-by-sample against SwinYNet, a Swin Transformer-based multi-task model trained end-to-end on FRB data for joint detection, segmentation, and parameter estimation. Formally, the specialized detector optimizes \((\theta^\star, \varphi^\star) = \arg\min_{\theta,\varphi} \mathcal{L}_{\text{FRB}}\), while the VLM uses fixed \(\theta\) from general pretraining with a prompt-induced task head: \(\hat{y} = g_P(f_\theta(x))\).

Key Contributions

  • First systematic zero-shot, prompt-only benchmark of small local VLMs against a state-of-the-art specialized FRB detector on a controlled simulated dataset.
  • Demonstration that Gemma 4 2B achieves accuracy statistically equivalent to SwinYNet (93.65% vs. 92.90%) without any domain-specific training.
  • Evidence that zero-shot VLMs exhibit substantially lower false-positive rates on structured RFI (6.4% vs. 25.0%) and zero false positives on pure noise.
  • Prompt-only reconfiguration to three-class FRB/RFI/noise classification reaching up to 86% accuracy with no false FRBs, requiring only a prompt rewrite.
  • Practical case for small, open-weight, locally-executable VLMs as a data-efficient and cost-accessible baseline for radio transient screening.

Results

  • Binary classification accuracy: Gemma 4 2B: 93.65%; SwinYNet: 92.90% — no statistically significant difference.
  • ROC-AUC: SwinYNet: 1.0000; Gemma 4 VLMs: 0.9482 — SwinYNet retains perfect probabilistic ranking.
  • False-positive rate on structured RFI: Gemma 4 2B: 6.4% vs. SwinYNet: 25.0%.
  • False positives on pure noise: Gemma 4 2B: 0 (no false positives).
  • Three-class (FRB/RFI/noise) accuracy: up to 86% on the full 3000-sample set, with zero false FRB predictions, via prompt rewrite alone.

Limitations

  • Benchmark uses only simulated data; real observational data introduce label ambiguity, instrumental artifacts, and RFI complexity not captured by simulateSearch.
  • FRB prevalence in the benchmark (50%) is orders of magnitude higher than in real surveys, so reported metrics do not translate directly to operational false-positive rates.
  • SwinYNet's ROC-AUC of 1.0000 on this benchmark demonstrates a decisive advantage in probabilistic calibration and threshold-free ranking that zero-shot VLMs do not match.
  • Natural-language justifications produced by the VLM are inspectable but not physically causally verified; a plausible explanation may accompany an incorrect decision.
  • Only two model sizes tested (Gemma 4 2B and 4B); inference latency and throughput at survey scale are not fully characterized.
  • Zero-shot regime is a lower bound; the paper does not evaluate few-shot, RAG, or LoRA adaptation that would likely improve performance further.

Relevance to Vision-Language Models

This paper provides a rare controlled ablation of zero-shot VLM visual transfer on scientific imagery that is entirely outside the distribution of standard pretraining corpora, directly probing what general-purpose visual representations capture about structured physical signals. The result—near-parity with a supervised specialist at the default threshold, combined with superior RFI rejection—strengthens the case that the connector-architecture VLM (vision encoder \(E_v\) + projector \(W_v\) + autoregressive decoder) generalizes to domain-shifted image classification without any weight updates. For the VLM community, it contributes a reproducible benchmark in an underexplored scientific domain and empirically validates prompt-based task reconfiguration (binary → multiclass by prompt rewrite) as a zero-cost capability. It also highlights the persistent gap in probabilistic calibration (ROC-AUC 0.9482 vs. 1.0000), a known weakness of prompt-induced classifiers compared to dedicated softmax heads, motivating future work on calibrated structured output from VLMs.