Skip to content

Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration

🕒 Published (v1): 2025-12-04 04:34 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoDW is an agentic framework for automating complex, multi-step document workflows via stepwise API planning with adaptive dual-level rollback. It achieves 90% instruction-level and 62% session-level completion rates on a new benchmark, outperforming the best baseline by 40% and 76% respectively. The key insight is that incremental, state-conditioned planning combined with argument- and API-level fault recovery dramatically reduces cascading errors in long-horizon document sessions.

Problem

Existing LLM-based document automation agents either plan upfront (no error recovery), rely on simple retrieval (no state awareness), or require human-in-the-loop verification. All fail in multi-step, session-level workflows where a single erroneous API call propagates silently and derails subsequent steps—a problem that worsens with workflow length (sessions averaging 34.8 API calls).

Method

AutoDW orchestrates document workflows through three integrated modules:

  1. Stepwise planning: Generates one API call at a time, conditioned on the current user instruction, a fine-tuned 178M BERT intent classifier (8 classes, 98% accuracy) that narrows candidate APIs to top-3 intents, and the live document state extracted after each execution.

  2. API execution & state tracking: Runs each API in a Python runtime and extracts a seven-component document state tuple (paragraphs, tables, images, page layout, styles, interactive elements, document info). State parsing failure is itself treated as an execution error, triggering rollback.

  3. Adaptive rollback: An LLM validator receives a structured delta ∆ = (∆S, ∆C, ∆F, ∆St, ∆T, ∆H) between pre- and post-execution states and returns a binary pass/fail, a confidence score, and a textual explanation. If confidence ≥ 0.6 and decision is "fail", rollback fires in two escalating steps: (a) argument-level rollback regenerates arguments for the same API using the validator's explanation; if that also fails, (b) API-level rollback selects an entirely different API. A single round of dual-level rollback is used by default.

Key Contributions

  • AutoDW framework combining one-at-a-time state-conditioned API planning with dual-level (argument + API) adaptive rollback for document workflow automation.
  • DWBench: a bilingual benchmark of 250 sessions, 1,708 human-annotated instructions, 74 APIs, averaging 34.8 API calls per session (range 15–75); operational correctness evaluated by an LLM judge validated against human annotations.
  • State-of-the-art results, outperforming the next-best baseline by +40.1% instruction-level and +76.1% session-level accuracy.
  • Ablation showing rollback alone contributes a ~74% relative improvement in session-level success (35.6% → 62%), and that single-round dual-level rollback is the cost-optimal configuration.

Results

  • AutoDW vs. Hybrid (best baseline): 90.33% vs. 64.46% iACC (+40.1%); 62.00% vs. 35.20% sACC (+76.1%), at +26.5% token cost.
  • vs. Reasoning-only: 90.33% vs. 39.93% iACC; 62.00% vs. 25.20% sACC.
  • vs. Retrieval-only: 90.33% vs. 13.84% iACC; 62.00% vs. 4.40% sACC.
  • Backbone LLM robustness: Gemini-2.5-Pro (91.79% iACC / 67.2% sACC), GPT-4.1 (92.03% / 65.2%), DeepSeek-v3 (90.33% / 62.0%), Qwen-Plus (82.82% / 53.6%).
  • Difficulty robustness (with three stronger LLMs): hard instructions (≥6 APIs) achieve iACC above 86%, only 4.4% below overall.
  • Ablation — rollback impact: wo. rollback → arg. rollback → single-round (AutoDW) → two-round: iACC 71.8% / 76.8% / 90.3% / 91.4%; sACC 35.6% / 43.6% / 62.0% / 64.8%. Two-round adds only +1.1% iACC / +2.8% sACC over single-round.

Limitations

  • Session-level accuracy remains at 62%, meaning ~38% of multi-turn sessions fail end-to-end, likely due to error accumulation over very long sessions (up to 75 API calls).
  • Only covers Microsoft Word (.docx) document automation; not evaluated on spreadsheets, PDFs, or presentations despite PPTC being a related benchmark.
  • Rollback is bounded to a single round by default; the system cannot recover from sequences of consecutive failures or structurally ambiguous instructions.
  • The LLM-based validator is itself fallible; low-confidence scores default to "accept," which may propagate subtle misalignments.
  • BERT intent classifier was trained on 3,315 examples covering only 8 fixed intents, limiting generalization to novel document operation types.
  • Token cost scales roughly linearly with rollback rounds; production deployment at high volume may be expensive.

Relevance to Agentic AI / LLM Agents

AutoDW directly addresses the core reliability challenge in tool-using LLM agents: cascading errors in long-horizon action sequences. The dual-level rollback mechanism—separating argument correction from API replacement—is a concrete fault-recovery primitive applicable beyond document workflows to any agentic system with API-level action spaces. The finding that session-level accuracy collapses from 90% to 62% (instruction → session) quantifies exactly how error propagation degrades end-to-end agent performance, motivating further work on stateful error recovery. DWBench, with its human-annotated API sequences and operational-correctness evaluation protocol, provides a rigorous methodology for benchmarking long-horizon agent reliability that complements GUI-based and web-navigation benchmarks like WebArena.