GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection

🕒 Published (v1): 2026-05-27 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GUI-CIDER is a three-stage mid-training framework that injects explicit GUI world knowledge into multimodal agents by synthesizing causal and planning knowledge from raw trajectories, filtering it with a density-aware exemplar reselection function, and training via next-token prediction. It bypasses the implicit, annotation-driven knowledge absorption of SFT/RL and eliminates the runtime overhead of multi-agent scaffolding. On a 100M-token synthesized corpus, an 8B model reaches near Claude-Sonnet-4.5-level GUI knowledge understanding.

Problem

GUI agents trained with SFT or RL only absorb world knowledge implicitly through action labels or reward signals, producing trajectory memorization rather than genuine comprehension. Multi-agent scaffolding can compensate but imposes significant runtime overhead. No prior open work explicitly internalizes GUI operation knowledge via mid-training.

Method

GUI-CIDER operates in three sequential stages:

Stage 1 — Data Synthesis. Given a trajectory dataset \(\mathcal{D} = \{\tau_1, \ldots, \tau_N\}\) where each \(\tau\) contains a task instruction \(T\) and a sequence \(\{(s_i, a_i)\}_{i=1}^{L}\), two knowledge types are distilled into purely textual samples: - Static planning knowledge: a high-capacity LLM decomposes task \(T\) into a sub-goal set \(S = \{g_1, \ldots, g_n\}\) via zero-shot hierarchical reasoning. - Dynamic causal knowledge: per-step actions \(a_t\) are grounded into semantic descriptions via \(B(a_t, v_t) \to a_t^{nl}\); screenshots \(s_{t-1}, s_t\) are converted to text descriptions \(d_{t-1}, d_t\); a causal analyst \(C(\cdot)\) extracts a rationale \(R_t = \{d_{t-1}, d_t, Trig_t, Mech_t, CoT_t\}\) encoding trigger, UI mechanism, and chain-of-thought.

Stage 2 — Exemplar Reselection. A retention probability \(g(x)\) combines two signals: $\(g(x) = \frac{1}{1 + \alpha d(x)} + \lambda \cdot f(x) \cdot \left(1 - \frac{1}{1 + \alpha d(x)}\right)\)$ where \(f(x) = \tanh\!\left(\frac{K(x)}{\gamma}\right)\) is a causal saliency score based on causal-logic token count, and \(d(x)\) is a min-max normalized relative K-NN density score penalizing redundant samples. Each sample is retained if \(\xi_x \sim \text{Uniform}(0,1) \le g(x)\).

Stage 3 — Mid-training. The filtered corpus \(X_{\text{high}}\) is serialized as plain text and trained with the standard causal LM loss: $\(\mathcal{L}_{\text{mid}} = -\sum_{x \in X_{\text{high}}} \sum_{i=1}^{L_x} \log P_\theta(y_i \mid y_{<i})\)$ No input/output distinction is made; the entire sequence is autoregressive. The model base is a general-purpose VLM (Qwen3-VL-4B/8B-Instruct), not a GUI-specialized model.

Key Contributions

• Mid-training paradigm for GUI agents that explicitly encodes GUI world knowledge rather than relying on implicit absorption via SFT/RL.
• A two-component data synthesis pipeline producing static planning knowledge and dynamic causal transition rationales from existing GUI trajectory datasets.
• A theoretically grounded retention function \(g(x)\) satisfying four provable properties: causal monotonicity, density penalty, density order preservation, and density-causal synergy.
• A synthesized corpus of ~100M tokens released for community use.
• Empirical finding that general-purpose base models are better mid-training targets than GUI-specialized models that have already been heavily post-trained.

Results

• Task completion (Table 2): GUI-CIDER improves step-wise success rate (SR) over zero-shot Qwen3-VL-8B: AITZ 56.91→60.70, AndroidControl high 52.49→54.09, GUI-Odyssey 44.16→48.55. Combined mid-training + SFT yields further gains; a 4B model after both stages surpasses the zero-shot 8B baseline.
• GUI knowledge (Table 4): GUI-CIDER-8B scores 66.51 overall on GUI Knowledge Bench, matching Claude-Sonnet-4.5 (66.53); it outperforms o3 (73.30), Gemini-2.5-Pro (71.69), and GPT-5-Chat (70.97) on the objective subset (71.81 vs. 69.45/67.72/68.98).
• GUI understanding (Table 3): GUI-CIDER-8B achieves 94.69 overall on MMBench-GUI L1, well above InternVL3-72B (79.15) and GPT-4o (60.16) at easy/medium/hard levels.
• Average relative improvement: 9.70% in task success rate (TSR) over post-training baselines.
• Ablation: Removing exemplar reselection causes substantial SR drop on GUI-Odyssey (e.g., 43.45→41.06 for 4B).

Limitations

• LoRA-only training due to compute constraints; full-parameter mid-training not explored.
• Experiments limited to 4B–8B parameter scales; larger models untested.
• Data synthesis relies on an expert LLM (additional inference cost not fully characterized).
• Mid-training corpus is dataset-specific for task completion benchmarks (not cross-dataset generalization).

Relevance to Harnesses / Meta-Harnesses

GUI-CIDER is directly relevant as an alternative to multi-agent scaffolding harnesses: it replaces runtime orchestration (calling a general agent to supply missing knowledge) with a data synthesis pipeline that bakes scaffolding knowledge into model weights at mid-training time. The three-stage pipeline (synthesize → filter → train) is itself a meta-harness over existing GUI trajectory datasets, operating as a knowledge distillation harness that converts raw agent trajectories into structured causal rationales. The finding that general-purpose models absorb mid-training knowledge better than heavily post-trained specialist models has direct implications for meta-harness design: excessive specialization at the post-training stage degrades a model's capacity to incorporate new knowledge injected by outer training loops.