Skip to content

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

🕒 Published (v1): 2026-07-20 17:12 UTC · Source: Arxiv · link

Why this paper was selected

Agent harness for real-time multimodal deployment; directly actionable for harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FlashRT is an agent harness that automates the conversion of single-GPU reference implementations of real-time multimodal pipelines into optimized multi-GPU deployments. It introduces a chain-of-program paradigm — an intermediate representation (IR) plus a self-driven validation loop — to overcome the two key failure modes of naive coding agents. Across video world models, voice agents, and multimodal LLMs, it achieves up to ~70× latency reduction and 3.6× throughput improvement with no human intervention beyond providing the reference code.

Problem

Efficiently serving real-time multimodal applications (voice agents, video generation, multimodal LLMs) requires application-specific deployment decisions — placement, streaming, intra-model parallelism — that existing systems cannot automate. Rule-based systems commit to fixed deployment policies and workload assumptions, auto-parallelism compilers (FlexFlow, Alpa) are calibrated to dense DNN training, and operator-level compilers (TVM) do not touch placement or scheduling. The problem of finding an optimal deployment is NP-hard even in simplified single-batch settings (reduces to non-preemptive multiprocessor scheduling), so no fixed policy suffices across diverse applications.

Method

FlashRT guides a generic coding agent through a structured three-phase workflow:

1. Chain-of-program IR construction. The agent transforms the reference implementation into a hierarchical directed acyclic graph IR. IR nodes encapsulate computation regions at multiple granularities (top-level application graph + nested intra-module graphs). Nodes carry persistent-state annotations (reads/writes, recovering cross-batch \(\lambda=1\) dependencies, i.e., nodes sharing a KV cache must be ordered across batches). Edges are labeled blocking or streaming (streaming edges mark producer–consumer overlap opportunities). An IR interpreter validates IR correctness by comparing sequential execution against the reference on the same inputs.

2. Static analysis. Tool-assisted static analysis over the IR identifies candidate transformations: parallelizable nodes (disjoint persistent state → eligible for disaggregation/pipeline parallelism) and streaming opportunities (streaming-labeled edges → eligible for cross-batch pipelining).

3. Measurement-gated optimization loop. Starting from the candidate list, the agent maintains a self-evolving variant queue. Each iteration: (a) states a hypothesis naming the transformation and target bottleneck, (b) implements it in isolation, (c) runs an agent-authored test harness that writes simulated inputs to the backend's input buffers and reads from output buffers (mimicking real user interaction) to verify element-wise output equivalence and measure end-to-end latency/throughput, and (d) enqueues new hypotheses based on results.

Key Contributions

  • Chain-of-program paradigm: structured IR intermediary that forces agents to reason at heterogeneous granularities before generating deployments, analogous to compiler lowering passes.
  • Hierarchical IR design with persistent-state and streaming annotations that deterministically surface disaggregation and pipeline parallelism opportunities a naive agent misses.
  • Application-grounded validation loop: agent-authored test harnesses grounded in simulated user I/O, enabling application-specific correctness and benchmark measurements without a real frontend.
  • Self-evolving variant queue: expands exploration diversity beyond any single optimization axis, allowing composition of strategies (e.g., inter-module streaming combined with model-level pipeline parallelism).
  • Cross-platform generalization: a single agentic framework achieves strong results on both NVIDIA B200 and AMD MI355X without hardware-expert tuning.

Results

  • Up to ~70× latency reduction across evaluated multimodal applications on NVIDIA B200.
  • 2.8× throughput improvement on NVIDIA B200; 3.6× throughput improvement on AMD MI355X.
  • For Qwen3-Omni text-to-audio inference on AMD MI355X: 65% response-latency reduction vs. the expert vLLM-Omni implementation.
  • Evaluated applications include video world models, live avatar (ASR → LLM → TTS → S2V pipeline), and multimodal LLMs.
  • AMD MI355X results match or exceed NVIDIA B200 peak latency reduction while yielding higher peak throughput gain (3.6× vs. 2.8×), attributed to lower baseline expert-optimization maturity on AMD.

Limitations

  • Input must be a synchronous, single-GPU Python reference implementation; applications without a clean reference are out of scope.
  • The IR generation, static analysis, and validation loop depend on the underlying coding agent's capability; failure modes not fully eliminated, only reduced.
  • Evaluation is restricted to a small set of applications; generalization to arbitrary multimodal pipelines is asserted but not exhaustively demonstrated.
  • No ablation isolating the contribution of each component (IR vs. validation loop vs. variant queue) beyond the case study analysis.
  • The paper text is truncated; complete quantitative comparisons against all baselines are not fully visible.

Relevance to Vision-Language Models

FlashRT directly targets multimodal LLMs — a class that includes VLMs — as one of its primary deployment targets, demonstrating optimized serving for models like Qwen3-Omni. The system's ability to decompose heterogeneous pipelines (vision encoder → language model → decoder) at adaptive granularities and discover intra-model parallelism (e.g., disaggregated DiT and VAE, pipeline-parallel diffusion steps) is directly applicable to VLM inference pipelines that interleave visual encoding, cross-modal attention, and text generation. For researchers tracking VLMs, FlashRT represents a direction where agent-driven system engineering replaces manual optimization, potentially enabling low-latency deployment of new VLM architectures without per-model systems work. It also connects to real-time vision-language-action policies and video world models, both of which embed VLMs as core components.