Skip to content

FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting

🕒 Published (v1): 2026-07-16 12:23 UTC · Source: Arxiv · link

Why this paper was selected

Evidence-guided multi-agent recovery harness; fault-tolerance patterns for fragile tool environments

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FirmPilot is an evidence-guided multi-agent framework that reformulates IoT firmware rehosting as iterative, execution-validated environment reconstruction. Five specialized agents (SEARCH, PLAN, FILE, NVRAM, NETWORK) coordinate through typed action interfaces grounded in runtime observations and RAG-retrieved evidence. On 10,033 real-world firmware images, it more than doubles web-service reachability over the FirmAE baseline (25.49% → 52.39%).

Problem

Whole-system firmware rehosting in QEMU fails on heterogeneous real-world corpora because success depends on tightly coupled, cross-layer dependencies: boot/init scripts, NVRAM-backed persistent configuration, interface naming, and service launch order. Template-driven pipelines (e.g., FirmAE) cannot cover long-tail vendor conventions, while unconstrained LLM operators risk unsupported artifact mutations and irreproducible executions. Existing guidance systems expose evidence without closing the loop into bounded environment transitions.

Method

FirmPilot models rehosting as a bounded transition system. At iteration \(t\), artifact state \(x_t\) (filesystem overlays, NVRAM overrides, network scripts) is executed in QEMU, yielding observations \(o_t\) (serial logs, ICMP/HTTP probes). The orchestrator maintains context \(c_t = \langle x_{\leq t}, o_{\leq t}, E_{\leq t} \rangle\) and applies an ordered plan \(p_t\) of typed delta functions:

\[x_{t+1} \leftarrow x_t \oplus \bigoplus_{a \in p_t} \delta_{t,a}\]

Five agents have strictly bounded action surfaces: - SEARCH: similarity-based RAG over Web/GitHub (vendor docs, forums, repos); read-only, no artifact mutation; admits only top-ranked cited snippets with provenance metadata. - PLAN: schedules FILE → NVRAM → NETWORK in dependency order; never writes artifacts directly; observes updated logs after each round. - FILE: recovers init entrypoints and service-launch specs; path-validates and filters shell metacharacters; emits replayable startup deltas. - NVRAM: per-firmware key–value inference from runtime reads, default files, and config strings; LLM resolves only unresolved priority keys; overlays isolated from rootfs. - NETWORK: maps noisy logs to typed IP/interface/bridge/VLAN/port/QEMU-argument tuples; accepted only after re-execution confirms both ICMP reachability and a live HTTP(S) endpoint.

A transition is accepted only after re-execution and fixed probe confirmation; the loop terminates on web reachability, budget exhaustion, or no admissible actions. The LLM backend throughout is DeepSeek V4 Flash; each image receives a 2,400-second wall-clock budget with at most three planner rounds.

Key Contributions

  • Formalization of firmware rehosting as evidence-indexed environment reconstruction under partial observability, with execution-accepted typed transitions resolving coupled boot/state/network dependencies.
  • FirmPilot system design: evidence-guided multi-agent loop with per-role bounded action interfaces for filesystem/init recovery, NVRAM state synthesis, and network exposure recovery.
  • Large-scale evaluation on 10,033 LFwC images with full-corpus ablations, a general-purpose coding-agent baseline (Claude Code), and downstream workflow analysis (service discovery, RouterSploit, protocol-aware fuzzing).

Results

  • LFwC corpus (10,033 images): network reachability 39.30% (FirmAE) → 71.93% (FirmPilot); web-service reachability 25.49% → 52.39% (+2,699 absolute images).
  • Public FirmAE benchmark (1,122 images): web-service reachability 79.4% → 82.2% (smaller gap; templates already cover common cases).
  • Average detected services per firmware: 0.86 (FirmAE) → 1.62 (FirmPilot).
  • General-purpose coding-agent baseline (Claude Code): only 5.43% web-service reachability on LFwC, confirming domain-specific coordination is necessary.
  • Downstream workflows on 5,256 web-reachable rehosts: nmap service discovery, 193 manually validated RouterSploit findings, and protocol-aware fuzzing over recovered service surfaces.
  • Vendor-level gains are large for Linksys (0% → 53.18% web), Ubiquiti (0.93% → 28.41%), AVM (5.32% → 12.96%), and EnGenius (17.93% → 80.69%); D-Link and AVM remain lower in absolute terms.

Limitations

  • Evaluation LLM backend is DeepSeek V4 Flash; results may not generalize to other model choices.
  • Maximum of three planner rounds within the 2,400-second budget limits recovery depth for highly complex firmware.
  • D-Link, Ubiquiti, and AVM remain substantially below 50% web-service reachability, indicating unresolved vendor-specific failure modes.
  • SEARCH agent relies on public Web/GitHub retrieval; proprietary or obscure vendor documentation is not accessible.
  • No coverage of non-Linux firmware, RTOS, or bare-metal targets; restricted to whole-system QEMU OS emulation.
  • The paper is truncated; full ablation breakdowns (per-component contribution magnitudes) are not completely visible.

Relevance to Harnesses / Meta-Harnesses

FirmPilot is a concrete instance of a domain-specific multi-agent meta-harness: it wraps an underlying execution substrate (QEMU + FirmAE-style pipeline) with an orchestration loop that coordinates heterogeneous specialized agents, enforces typed action interfaces, and gates state transitions on empirical probe outcomes rather than model confidence. The design pattern — PLAN agent as loop controller dispatching to bounded specialist agents, with SEARCH providing retrieval-augmented grounding — directly parallels harness architectures that separate orchestration logic from execution substrates. The contrast with the unconstrained Claude Code baseline (5.43% vs. 52.39%) is a sharp empirical argument that domain-constrained harness structure, not raw LLM capability, drives reliability in iterative agentic loops. Researchers tracking harness design should note the formal transition-system framing and the per-role action-surface bounding as reusable patterns for any iterative, execution-validated agentic workflow.