Skip to content

APD-Agents: A Large Language Model-Driven Multi-Agents Collaborative Framework for Automated Page Design

🕒 Published (v1): 2025-11-18 03:39 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

APD-Agents is a five-agent LLM-driven pipeline for automated mobile UI layout generation that produces editable structured JSON rather than static images. An OrchestratorAgent schedules four specialized agents to generate layouts coarse-to-fine, using semantic retrieval for few-shot in-context learning. The system achieves state-of-the-art mIoU and overlap scores on RICO without task-specific training.

Problem

Existing layout generation methods (LayoutGAN, LayoutTransformer, LayoutDiffusion, etc.) output non-editable static images incompatible with tools like Sketch/Figma; struggle with complex pages containing many heterogeneous elements; require large domain-specific annotated datasets that generalize poorly across design specifications; and cannot dynamically adapt to diverse user requirements expressed in natural language.

Method

The framework orchestrates five LLM agents in a fixed task graph:

  1. OrchestratorAgent — ingests user's natural-language description, dispatches subtasks, aggregates outputs into final JSON.
  2. SemanticParserAgent — extracts an element hierarchy tree with class \(c_i\), position \((x_i, y_i)\), size \((w_i, h_i)\), and value \(v_i\).
  3. TemplateRetrievalAgent — embeds element descriptions via GLM-embedding-v2, selects top-\(k\) templates by cosine similarity from a pre-cached library for few-shot ICL.
  4. PrimaryLayoutAgent — generates coarse first-layer bounding boxes guided by retrieved exemplars and LLM-summarized design specifications.
  5. RecursiveComponentAgent — performs depth-first recursive expansion of composite nodes (e.g., Toolbar, Button), querying TemplateRetrievalAgent at each level until all nodes are basic elements.

Output is a hierarchical JSON matching the target format directly importable by design software. The system is built on AutoGen with Doubao-1.5-pro-32k at temperature 0.7.

Key Contributions

  • Multi-agent orchestration pipeline decomposing layout generation into parse → retrieve → coarse layout → recursive refinement stages.
  • Coarse-to-fine generation that avoids LLM token-limit failures on complex pages (one-go had 18% failure rate on 600 test cases).
  • Training-free design-specification summarization: design rules are dynamically extracted from retrieved templates via LLM rather than hard-coded.
  • New metric EPAcc that compares generated and ground-truth layout trees node-by-node across class, text, style, and hierarchy.
  • Structured JSON output directly loadable by Sketch/Figma.

Results

  • mIoU: 0.485 vs. 0.440 (LST-none, best baseline) — +0.045 improvement.
  • Overlap (Ovp): 0.448 vs. 0.491 (LayoutDiffusion, best baseline) — −0.043 reduction.
  • Alignment (Ali): 0.192 vs. 0.100 (LayoutTransformer, best baseline) — slightly worse.
  • EPAcc: 85.90%; class omissions = 0%; >97% of remaining errors are style/key mismatches.
  • Coarse-to-fine vs. one-go (600-page ablation): mIoU 0.471 vs. 0.407; EPAcc 85.58% vs. 65.83%; one-go produced 110 failures (18%).

Limitations

  • Alignment (Ali) metric lags LayoutTransformer by 0.092; multi-step recursive generation introduces spatial inconsistency across levels.
  • Style errors persist when the template library lacks coverage for specific element types (icons, text buttons); element style is underspecified by type alone.
  • Coarse-to-fine increases token consumption and latency relative to one-shot methods; design-specification summarization adds additional LLM calls per page.
  • Retrieval pool capped at 500 pages per generation call; performance with very small template libraries is not fully characterized.
  • Evaluated only on RICO (Android apps); cross-domain generalization is asserted but not measured.

Relevance to Harnesses / Meta-Harnesses

APD-Agents is a textbook example of a domain-specific meta-harness: a fixed orchestration layer (OrchestratorAgent) that schedules, sequences, and aggregates outputs from purpose-built sub-agents with defined roles and interfaces, built on a general agent framework (AutoGen). The coarse-to-fine recursive dispatch pattern — where each agent can re-invoke the retrieval service at multiple abstraction levels — mirrors the hierarchical task decomposition common in general-purpose harness designs. The explicit agent-role separation (parse, retrieve, generate, refine) demonstrates how harness authors can enforce modularity and reuse across pipeline stages. This paper offers a concrete instantiation of the orchestrator-specialist pattern applicable to any structured generation domain.