PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series in Typhoon Forecasting

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PIPE introduces physics-informed positional encoding to embed temporal and geospatial metadata (timestamps, latitude, longitude) into VLMs for multimodal time series forecasting. Applied to typhoon track and intensity prediction using the Digital Typhoon satellite dataset, PIPE-3B achieves state-of-the-art results over both domain NWP models and general-purpose VLMs. The key insight is that existing position encodings only capture intra-instance token order, missing the cross-instance global physical knowledge encoded in satellite imagery metadata.

Problem

Existing multimodal time series forecasting methods align text instructions with numerical sequences but ignore visual modalities (e.g., satellite imagery). When VLMs like GPT-4o or Qwen-VL are applied, their vision encoders (CLIP, Q-Former, MLP projectors) extract only pixel-level semantics and discard the rich physical metadata — timestamp, latitude, longitude — embedded in each image. Standard positional encodings (sinusoidal, RoPE) only model intra-instance sequential order and cannot encode global cross-instance spatiotemporal relationships, leading to poor multimodal alignment for geophysical forecasting tasks.

Method

PIPE modifies the position encoding pipeline of a VLM (Qwen-2.5-VL-3B with 3D RoPE) with two components:

1. Physics-Informed Positional Indexing: Vision token position IDs are set to physically meaningful values rather than sequential indices. Temporal IDs encode t = tday × 24 + thour (hours within a year, range 0–8784); height IDs encode latitude (0–180); width IDs encode longitude (0–360). To prevent overlap with text token IDs (which use standard sequential indexing 0 to seq_len), all vision position IDs are mapped to negative values. RoPE is then applied on these IDs to encode intra-instance relationships.

2. Variant-Frequency Positional Encoding: Standard sinusoidal PE is modified by introducing a variable period p specific to each physical dimension (pday=366, phour=24, platitude=180, plongitude=360). The encoding PE(pos, 2i) = sin(pos/p × 1/10000^(2i/d)) maps each physical variable into a distinct frequency band in the embedding space. Text tokens retain standard sinusoidal encoding for pretrained LLM compatibility. These PEs are added to input embeddings before the decoder stacks.

The base model is Qwen-2.5-VL-3B fine-tuned via LLaMA-Factory; vision encoder is frozen; LLM layers are trained. The task is forecasting intensity (hPa), latitude, and longitude of typhoons given 12h of historical satellite images + numerical track data.

Key Contributions

• Formalizes multimodal time series forecasting as a task pairing numerical sequences with corresponding per-timestep satellite images, extending beyond text-augmented forecasting.
• Physics-informed positional indexing: maps real-world temporal (hourly-in-year) and geospatial (lat/lng) coordinates to RoPE position IDs, with negative-value remapping to eliminate text/vision ID overlap.
• Variant-frequency sinusoidal encoding: per-variable period scaling to encode frequency characteristics of physical variables directly into input embeddings.
• State-of-the-art on Digital Typhoon dataset (1,116 sequences, 192,956 images, 40+ years) across intensity, latitude, longitude, and track distance metrics.

Results

• vs. best unimodal AI baseline (TiDE): 12% MAE improvement for intensity at 6h lead time (1.515 vs. 1.724 hPa MAE); track distance error 16.275 km vs. 34.068 km.
• vs. Qwen-2.5-VL-3B (same base, no PIPE): intensity MAE 1.515 vs. 1.617; longitude MAE 0.095 vs. 0.103; track distance 16.275 vs. 17.129 km.
• vs. Gemini-2.5-flash (zero-shot): intensity MAE 1.515 vs. 1.924; track distance 16.275 vs. 36.006 km.
• vs. domain models (ECMWF, PanGu, GenCast): PIPE-3B outperforms on track distance where available (PanGu: 32.892 km at 12h; PIPE-3B: 16.275 km at 6h — note different lead times; direct comparison limited by availability).
• Ablation (6h lead):
• Removing vision: intensity MAE +8% (1.646 vs. 1.515).
• Replacing physics-informed indexing with 3D indexing: +6% intensity MAE degradation.
• Removing negative indexing: catastrophic overlap causes distance error jump to 53.548 km.
• Removing variant-frequency PE: track distance 16.860 vs. 16.275 km.
• Australian region (zero-shot generalization): PIPE-3B zero-shot beats tuned Qwen-2.5-VL-3B on intensity RMSE (2.773 vs. 2.806) and distance (19.090 vs. 19.718 km).

Limitations

• Processing high-resolution satellite images substantially increases computational complexity, constraining input sequence length and forecasting horizon.
• Physical laws and domain constraints (e.g., thermodynamic principles governing typhoon intensification) are not encoded — only metadata coordinates are injected; interpretability remains limited.
• RMSE for intensity at 6h is marginally worse than TiDE and PatchTST (2.981 vs. 2.819/2.867), suggesting the approach does not universally dominate on all metrics.
• Evaluation is restricted to one primary dataset (Western North Pacific typhoons, Digital Typhoon) and one regional generalization test (Australian basin); broader climate tasks untested.
• Domain model comparisons (ECMWF, PanGu, GenCast) use numbers from their original papers at potentially different lead times/settings, making direct comparison approximate.

Relevance to Vision-Language Models

PIPE directly addresses a gap in VLM adaptation: general-purpose VLMs (GPT-4o, Qwen-VL, Gemini) are tuned for pixel semantics but cannot exploit the physical metadata co-registered with scientific imagery. The work demonstrates that position encoding — typically an implementation detail — is a high-leverage intervention point for domain-grounding VLMs without modifying architecture. For VLM researchers, the negative-value remapping trick to prevent cross-modal positional ID collision is a practically important alignment lesson. More broadly, PIPE exemplifies a direction of lightweight physics-aware adaptation of VLMs for scientific/geospatial domains, complementary to instruction tuning and retrieval-augmentation approaches.