Skip to content

VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models

🕒 Published (v1): 2026-06-26 01:54 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CODA optimizes a planar meta-optic front-end's continuous density via adjoint-gradient FDTD to minimize the cross-entropy loss of a frozen CLIP classifier, replacing conventional focusing objectives with a downstream VLM objective. On simulated ImageNet-100 imaging, VLM-warm (200-iteration budget) improves CLIP ViT-L/14 zero-shot accuracy from 53.75% (Focus-opt baseline) to 65.41%, and transfers without re-optimization to SigLIP and DINOv2 on CIFAR-100 and Food-101.

Problem

Conventional optical design optimizes human-interpretable image-fidelity metrics (PSNR, SSIM, Strehl ratio, focal concentration). These criteria do not align with the representation space of frozen foundation VLMs, creating a mismatch under constrained meta-optic front-ends where compactness, weight, or manufacturability limits image quality. Prior end-to-end optics-AI co-design jointly optimizes optics and a task network, making it ambiguous whether gains come from the front-end or the adapted backend — leaving open the question: can a frozen VLM be better served by changing only the optical front-end?

Method

CODA optimizes only the continuous permittivity density \(\rho \in [0, 1]^N\) of a 2D planar meta-optic (5 µm × 600 nm, 13,400 variables; \(\varepsilon_{\min}=1\), \(\varepsilon_{\max}=5.76\)). For each of \(K=9\) wavelength–angle conditions \(\{(\lambda_k,\theta_k)\}\) (3 wavelengths × 3 angles), a 2D TE-polarized FDTD simulation (Meep) produces a 1D point-spread function (PSF) measured at the sensor line. A differentiable line-scan image-formation model convolves these PSFs row-wise with clean images to produce sensor tensors. The frozen VLM (CLIP ViT-L/14) consumes these sensor images; its cross-entropy loss is back-propagated through the VLM encoder and the image-formation model to the PSF interface. The gradient \(\partial \mathcal{L}/\partial \mathrm{PSF}_k\) becomes the source for an adjoint Maxwell solve, yielding \(\partial \mathcal{L}/\partial \rho_i = \mathrm{Re} \sum_k \omega_k^2 \varepsilon_0 \Delta\varepsilon \int_{\Omega_i} E_k \cdot E_k^{\text{adj}}\,d\mathbf{r}\) in 18 FDTD calls per iteration (vs. \(1.2\times 10^5\) for finite differences). DINOv2 uses a fixed linear probe (fit once on clean-image features). Three optical designs are compared: Focus-opt (focal-concentration objective \(\mathcal{L}_{\text{focus}}\), 200 iterations), VLM-cold (CODA from random initialization, 100 iterations), and VLM-warm (100 CODA iterations from a Focus-opt iteration-100 checkpoint).

Key Contributions

  • Formulates frozen-VLM optical front-end optimization as single-objective minimization of downstream classification loss without auxiliary image-fidelity losses.
  • Connects frozen-encoder automatic differentiation to adjoint-gradient meta-optic updates through a PSF backward interface.
  • Demonstrates +11.66 pp improvement over a model-agnostic focusing baseline (65.41 % vs. 53.75 % on ImageNet-100 with CLIP ViT-L/14).
  • Shows zero-shot transfer of the same optimized optics (no optical re-optimization) across CLIP, SigLIP, and DINOv2 on ImageNet-100, CIFAR-100, and Food-101, with VLM-warm beating Focus-opt in all nine dataset–encoder pairs (+7.61 to +20.80 pp).

Results

  • ImageNet-100 / CLIP ViT-L/14 zero-shot: Clean 88.26 %, Fresnel 8.10 %, Focus-opt 53.75±3.57 %, VLM-cold 47.87±1.76 %, VLM-warm 65.41±3.99 % (+11.66 pp vs. Focus-opt). The worst VLM-warm seed (61.44 %) exceeds the best Focus-opt seed (57.84 %).
  • Matched-budget robustness: VLM-warm also exceeds the best earlier Focus-opt checkpoint (iteration 50, 58.95±3.07 %) by +6.46 pp.
  • PSF metrics do not predict gain: Concentration, FWHM, peak position, and peak intensity tie or favor Focus-opt, yet VLM-warm has +11.66 pp higher accuracy. The main optical correlate is a 20 % larger intensity-weighted second moment (broader PSF base).
  • Linear probe on CLIP embeddings: VLM-warm 73.49±3.65 % vs. Focus-opt 63.45±2.57 % — gain persists without text prompts.
  • Transfer (VLM-warm vs. Focus-opt, all +): ImageNet-100: CLIP +11.66, SigLIP +13.97, DINOv2 +7.61 pp. CIFAR-100: CLIP +16.56, SigLIP +14.70, DINOv2 +20.80 pp. Food-101: CLIP +17.96, SigLIP +18.51, DINOv2 +20.80 pp.

Limitations

  • 2D FDTD + line-scan image formation, not full 3D wide-FOV simulation; field-dependent sensor geometry is not modeled.
  • Three discrete wavelengths (450/550/650 nm) mapped to RGB channels; achromatic or broadband design is outside scope.
  • All wavelength–angle PSFs share a common readout coordinate — no variation in sensor-line location.
  • DINOv2 evaluation uses a fixed clean-image linear probe (no zero-shot capability), not directly comparable to CLIP/SigLIP zero-shot.
  • Peak GPU memory 42 GB at batch size 16; scalability limits not explored.
  • Results are simulated; experimental fabrication and measurement are not performed.

Relevance to Vision-Language Models

This paper demonstrates that the visual front-end before a frozen VLM can be optimized for the VLM's own representation space, yielding substantial recognition gains under physically constrained optics. It concretely shows that aligning optical design with frozen VLM objectives outperforms conventional image-quality criteria, and that optics optimized for one VLM (CLIP) transfer to others (SigLIP, DINOv2) without re-optimization — suggesting a shared sensitivity in how these models process optically degraded inputs. For VLM researchers, it opens the possibility of co-adapting the physical acquisition hardware (meta-optics, sensors) to foundation model inference pipelines rather than the reverse.