LLMs as Firmware Experts: A Runtime-Grown Tree-of-Agents Framework¶

🕒 Published (v1): 2025-11-23 13:19 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

FirmHive is a recursive multi-agent framework (Tree of Agents, ToA) that enables LLMs to autonomously analyze firmware for security vulnerabilities. It replaces static orchestrator topologies with a runtime-grown agent tree where any agent can spawn sub-agents via a first-class delegation primitive. On the Karonte dataset it finds 1,802 vulnerabilities at 71% precision, outperforming both LLM-agent baselines and static SOTA tools.

Problem¶

Existing LLM agent architectures—monolithic ReAct agents, flat multi-agent systems (MAS), and centralized orchestrator-MAS—fail on large-scale firmware analysis because: (1) monolithic agents exhaust their context window over long reasoning chains; (2) flat MAS gain breadth but lose depth; (3) centralized orchestrators become bottlenecks as firmware complexity concentrates unbounded state in a single coordinator. No prior framework treats delegation as a per-agent capability that can be recursively exercised at runtime.

Method¶

FirmHive is built on two mechanisms:

Delegation as a built-in agent capability. Each agent receives a delegate(subtask) tool. A parent specifies an (object, target) task pair, calls the tool, and receives structured results—full context isolation (fresh LLM instance per sub-agent) with parent–child-only communication. Delegation supports two modes: parallel (1→n) for breadth and chained (1→1) for depth.

Runtime Tree of Agents (ToA). Starting from a single root agent, recursive delegation grows the tree at runtime. Tree shape (depth, branching factor, node count) is not pre-specified; it is determined by firmware structure and task complexity. Growth is constrained to three agent types (Directory, File, Function), hard recursion-depth and branching limits, and explicit per-agent termination conditions.

Recursive Delegation Engine (RDE) implements object-target decomposition: $$\text{Task}(o, g) \xrightarrow{\text{observe}} O \xrightarrow{\text{LLM},C} \{T_{\text{sub}}(o_i, g_i)\}_{i=1}^{k}$$ where context $C$ and observation $O$ are fixed by the firmware, constraining LLM variability to the delegation decision only (temperature=0, fixed seed).

Proactive Knowledge Hub (PKH) is a persistent global shared memory. Agents publish findings to PKH; PKH reconciles cross-branch results, recovers cross-component dependencies, and prevents redundant exploration—compensating for the lack of cross-branch messaging enforced by ToA constraints.

Key Contributions¶

Delegation as a first-class, per-agent capability rather than hardcoded orchestrator logic, enabling any agent to autonomously spawn and manage sub-agents.
Tree of Agents (ToA): a runtime-grown dynamic hierarchy whose depth and breadth adapt to firmware structure and task complexity.
Recursive Delegation Engine (RDE) with object-target task decomposition and controlled autonomy (stable anchors: context + observation).
Proactive Knowledge Hub (PKH) as a durable, queryable shared memory providing global coherence without cross-branch messaging.
First end-to-end autonomous LLM framework for firmware security analysis with no handcrafted rules or static pipelines; open-sourced at github.com/bjtu-SecurityLab/firmhive.
Evaluation across five firmware analysis tasks of increasing reasoning complexity (T1–T5: credential detection → version identification → config dataflow tracing → input tracing → full-stack vulnerability detection).

Results¶

Compared to LLM-agent baselines (Monolithic, MAS, MAS+Orchestrator) on the Karonte firmware dataset: - ~16× more reasoning steps (depth) - ~2.3× more files inspected (breadth) - ~5.6× more alerts per firmware image

Compared to SOTA static tools (Mango, SaTC): - 1,802 vulnerabilities identified autonomously - 71% precision—described as approaching expert-level analytical capability - Higher yield and higher fidelity simultaneously

Stability: repeated runs yield highly similar agent counts and results (structural convergence under temperature=0 + fixed seeds).

Limitations¶

Evaluation is limited to the Karonte dataset; generalizability to other firmware corpora is unverified.
Unconstrained delegation is dangerous; ToA correctness depends on the LLM reliably respecting recursion-depth and branching limits imposed only via prompt-level constraints, not hard runtime enforcement.
PKH reconciliation strategy and its scalability under very large firmware images are not fully detailed.
No ablation isolating the contribution of PKH versus ToA structure to the performance gains.
The paper is truncated before the File and Function agent descriptions, leaving part of the agent-type specification unclear.
LLM cost (token consumption at ~16× reasoning steps vs. baselines) is not quantified.

Relevance to Harnesses / Meta-Harnesses¶

FirmHive is a concrete instantiation of a meta-harness pattern: a runtime self-organizing harness where the orchestration topology is not pre-specified but grown dynamically by the tasks themselves. The Recursive Delegation Engine is effectively a meta-harness primitive—it defines how to spawn, configure, constrain, and collect from sub-agents, while the ToA is the emergent artifact. The PKH is analogous to a shared harness state bus. This is directly relevant to harness/meta-harness research because it demonstrates how replacing a static orchestration graph with a capability-driven recursive delegation primitive scales coordination to otherwise intractable problem sizes, and it provides concrete design constraints (task-context isolation, structural evolution limits, termination invariants) that any meta-harness must enforce to remain bounded and reproducible.