Adaptive Multimodal Agents-Based Framework for Automatic Workflow Execution¶

🕒 Published (v1): 2026-05-27 15:23 UTC · Source: Arxiv · link

Why this paper was selected

Adaptive multimodal agents for automatic workflow execution bridging metadata to perception

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a two-phase multimodal multi-agent framework for automatic GUI workflow execution: an offline discovery phase builds a topological knowledge graph from execution logs, and an online inference phase navigates that graph via Adaptive GraphRAG plus a closed-loop multi-agent verification protocol. Evaluated on GUIOdyssey (1,666 cross-app mobile tasks), the framework triples the Success Rate of the best competing method.

Problem¶

Existing GUI agents treat task sequences as isolated, linear episodes, failing to model the shared transition topology across tasks. This "tunnel vision" prevents cross-task knowledge reuse, makes agents brittle on novel/non-stationary interfaces, and causes context-loss failures (action hallucination, infinite loops) in multi-app workflows.

Method¶

Offline (Discovery) Phase: Raw execution logs are processed by a MLLM-based pipeline that classifies action tuples $(I_{\text{before}}, a, I_{\text{after}})$ into in-page vs. page-jump transitions. A dual-level similarity check (BGE-M3 semantic embeddings + visual comparison, indexed via FAISS with top-4 candidate retrieval) determines whether each new state maps to an existing graph node or creates one. Consecutive in-page operations are condensed into single edges. Only a stochastic $1/50$ subsample of training episodes is used for computational tractability.

Online (Inference) Phase: Given a task query $q$, the system retrieves the top-$k$ most similar traces from the graph knowledge base $K$ via cosine similarity: $$R(q, K) \rightarrow T_{\text{top-k}} = \arg\text{top-k}_{T \in K}\; \text{sim}(e_q, e_T)$$ A Context Builder synthesizes an Augmented Context via in-context learning. A hierarchical planning stack (Global Planner → Sub-goal Planner) decomposes the task, while an Observation Agent encodes the current GUI state. A Decision Agent proposes an atomic action, which a Verifier Agent evaluates against a consistency predicate: $$v(s_t, a_t, g_t) = \begin{cases} \texttt{APPROVE} & \text{if } C(s_t, a_t, g_t) = \text{True} \\ \texttt{REJECT}(f) & \text{otherwise} \end{cases}$$ Rejection triggers recursive sub-goal refinement for up to $M=4$ retries. History is maintained as a goal-aware differential state narrative rather than atomic action labels, preventing context loss and infinite loops.

Key Contributions¶

Two-phase pipeline separating offline topological knowledge construction from online graph-RAG-based navigation.
Automated workflow graph construction via MLLM-based state-jump classification and dual-level (semantic + visual) node deduplication.
Goal-aware differential state narrative for history, suppressing action hallucination and loop failures.
Closed-loop multi-agent Verifier that intercepts and validates proposed transitions before execution, with structured feedback propagation.
Empirical validation on GUIOdyssey (1,666 episodes, 5 domains); 3× SR improvement over best baseline.

Results¶

Benchmark: GUIOdyssey full test set, 1,666 cross-app mobile navigation episodes. Metrics: Action Matching Score (AMS) and Success Rate (SR).

The proposed framework triples the SR of the best alternative approach overall.
AMS is improved across all functional domains (Information Management, Web Shopping, Media Entertainment, Social Sharing, Multi-Apps) except Information Management, where PG-Agent (Qwen/72B) is superior.
PG-Agent (Qwen/72B) outperforms PG-Agent (Qwen/30B) on Information and Shopping but underperforms on Tool and Media, suggesting scale matters for information extraction tasks but not for general navigation.
Runtime: ~173 s/episode on 4×A100 (40 GB); per-step context 4,000–8,300 tokens, mean 6,200 tokens.

(Note: absolute AMS/SR numerical values for the proposed framework vs. baselines are referenced as "Table I" in the paper but the table itself is not included in the provided text.)

Limitations¶

Stochastic $1/50$ subsampling of training data for graph construction may under-represent rare transitions; graph completeness is not guaranteed.
The offline discovery phase requires a corpus of historical execution logs; cold-start on entirely new application domains is not addressed.
Verification loop bounded at $M=4$ retries; failure mode on exhaustion of retries is not deeply analyzed.
Absolute AMS/SR numbers for most baselines are not directly quoted in the provided text, making independent verification of the "3×" claim difficult.
Evaluation is limited to one benchmark (GUIOdyssey) focused on mobile cross-app navigation; generalizability to desktop or web environments is unvalidated.
Computational cost (~80 hours for 1,666 episodes on 4×A100) is high for production deployment.

Relevance to Harnesses / Meta-Harnesses¶

This framework is a concrete instance of a meta-harness pattern: an orchestration layer that coordinates multiple specialized sub-agents (Global Planner, Sub-goal Planner, Observation Agent, Decision Agent, Verifier Agent) through a structured pipeline rather than a monolithic model. The offline knowledge-graph construction phase is itself an automated harness that transforms raw execution traces into a reusable topological index, directly analogous to how meta-harnesses preprocess and index prior runs for downstream orchestration. The closed-loop Verifier-with-feedback loop mirrors the self-correcting retry logic common in agentic harness designs. For researchers tracking harness architecture, the two-phase (build-then-execute) separation and the multi-agent verification protocol are directly transferable patterns.