Skip to content

SPyCE: Skill-Policy Co-evolution for Multimodal Agents

🕒 Published (v1): 2026-07-15 14:01 UTC · Source: Arxiv · link

Why this paper was selected

RL skill-policy co-evolution for multimodal tool-using agents; addresses sparse reward problem

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPyCE introduces a closed-loop framework where rollout trajectories are distilled into a two-level hierarchical skill library (execution skills and workflow skills) that co-evolves with a GRPO-trained policy for multimodal tool-use agents. Unlike prior RL methods that discard trajectory content after computing scalar rewards, and unlike memory systems that never update the policy from stored experience, SPyCE creates a mutual feedback loop: better policies yield better skills, and better skills yield stronger policy priors. Evaluated across eight benchmarks, it consistently outperforms both RL-based and memory-based baselines.

Problem

Existing RL-based multimodal agents (DeepEyes, VTool-R1, CodeVision) reduce completed trajectories to scalar rewards, discarding the structured tool-use patterns they encode; reusable behaviors must be rediscovered from scratch each task. Memory-based alternatives (Reflexion, Voyager, Dynamic Cheatsheet, Agent-KB) store and retrieve past experience at test time but leave policy update decoupled from experience evolution — stronger policies do not automatically produce better memory, and better memory does not improve the policy. Furthermore, flat single-level memory cannot cleanly separate fine-grained local visual operations from high-level task workflow patterns, which is necessary for multimodal agents operating over heterogeneous visual evidence.

Method

SPyCE formulates multimodal tool-use as a POMDP and couples GRPO-style policy optimization with a hierarchical skill library through two tightly integrated stages.

Hierarchical Skill Library Construction. Successful trajectories \(\tau\) are distilled by an auxiliary MLLM (\(\text{MLLM}_{dk}\), here Qwen3-VL-235B) into two skill types: - Execution skills \(k = (c, u, e)\): condition–action–effect triples encoding when a local visual operation applies, what tool action to perform, and what evidence improvement to expect. - Workflow skills \(h = (b, w)\): pairs of a coarse bottleneck summary \(b\) and a workflow sketch \(w\) encoding multi-step tool orchestration patterns.

A merge-or-add consolidation rule prevents redundancy: a new candidate is merged into the nearest existing entry (by cosine similarity of trigger conditions for execution skills, bottleneck descriptions for workflow skills) if similarity exceeds threshold \(\theta_{\text{sim}}\); otherwise it is inserted as a new entry. Least-matched entries are pruned when the library exceeds capacity.

Skill-Policy Co-evolution. At every training step the policy \(\pi_\theta\) performs hierarchical retrieval: (1) generate a coarse bottleneck query from the task \((q, I)\) and retrieve top-\(K\) workflow skills; (2) conditioned on those priors, decompose into fine-grained local bottleneck descriptors and retrieve matching execution skills via \(k^{*(m)} = \arg\max_{k \in \mathcal{K}} \cos(\mathbf{E}_L(b^{(m)}), \mathbf{E}_L(c))\). The policy then acts as \(a_t \sim \pi_\theta(a_t \mid o_{\leq t}, q, I, H_T^{\text{top}}, \mathcal{K}_T^L)\). Every \(N\) policy steps, the skill libraries are refreshed from the top rollouts ranked by GRPO relative advantage \(A(\tau)\), with execution skills additionally pruned by empirical success rate \(\text{SR}(k) = N_{\text{success}}(k) / \max(N_{\text{retrieval}}(k), 1)\).

The reward penalizes redundant tool use on successful rollouts only: $\(R(\tau) = R_{\text{acc}}(\tau) - \lambda_{\text{call}} \cdot \mathbf{1}[\text{Succ}(\tau)=1] \cdot N_{\text{tool}}(\tau)\)$

Backbone models are Qwen3-VL-4B/8B-Instruct with SFT warm-start (5K CodeVision-SFT samples) followed by RL on ~12K filtered CodeVision-RL samples.

Key Contributions

  • First framework to establish an online closed co-evolution loop between a distilled skill library and a policy during RL training for multimodal agents.
  • Hierarchical skill abstraction separating execution skills (condition–action–effect triples for local visual bottlenecks) from workflow skills (bottleneck description + workflow sketch for global orchestration).
  • Merge-or-add consolidation with cosine-similarity deduplication and frequency-based pruning for controlled library growth.
  • Periodic online skill evolution using top-advantage rollouts to ensure skills track policy improvement, distinguishing SPyCE from static memory retrieval systems.
  • Demonstration that both hierarchy levels and online evolution are individually necessary (ablation studies, not shown in provided text).

Results

All figures are Task Success Rate (%) unless noted. Backbone: Qwen3-VL-4B-Instruct vs. 8B-Instruct.

4B model (vs. best comparable baseline in each column): - V* Bench: SPyCE 80.6 vs. GRPO 78.5 (+2.1), Agent-KB 76.4 (+4.2) - HRBench-4K: 78.8 vs. GRPO 76.0 (+2.8) - HRBench-8K: 73.9 vs. GRPO 70.4 (+3.5) - ChartQAPro: 51.4 vs. Agent-KB 50.4 (+1.0) - MathVerse: 55.0 vs. GRPO 49.5, Agent-KB 49.8 (+5.2) - MathVision: 57.4 vs. GRPO 52.9, Agent-KB 54.8 (+2.6) - WeMath: 59.7 vs. GRPO 55.9, Agent-KB 57.1 (+2.6)

8B model: - V* Bench: SPyCE 84.3 vs. GRPO 82.2 (+2.1) - HRBench-4K: 78.0 vs. GRPO 77.2 (+0.8) - HRBench-8K: 74.3 vs. GRPO 72.9 (+1.4) - ChartQAPro: 52.5 vs. Agent-KB 51.1 (+1.4) - MathVerse: 68.7 vs. GRPO 66.5, Agent-KB 65.7 (+3.0) - MathVision: 61.2 vs. GRPO 57.7, Agent-KB 56.6 (+4.6) - WeMath: 67.8 vs. GRPO 66.6, Agent-KB 65.8 (+2.0)

SPyCE achieves top rank on all seven reported benchmarks for both backbone sizes. (Table 2 / TIR-Bench agentic results are truncated in the provided text.)

Limitations

  • Skill distillation requires a very large auxiliary MLLM (Qwen3-VL-235B-A22B-Instruct), introducing substantial inference cost during training that is not borne by RL-only baselines.
  • The framework is evaluated only on visual tool-use tasks; generalization to non-visual agentic domains is unexamined.
  • Library consolidation and periodic evolution add hyperparameters (\(\theta_{\text{sim}}^K\), \(\theta_{\text{sim}}^H\), \(N\), \(\rho_{wf}\), \(\rho_{ex}\), library capacity) whose sensitivity is not fully characterized in the provided text.
  • The provided text is truncated; full ablation results, TIR-Bench numbers, and any stated limitations section are not available for review.
  • Embeddings for retrieval (text-embedding-3-small) may not optimally capture visual bottleneck semantics described in natural language.

Relevance to Agentic AI / LLM Agents

SPyCE directly addresses a core challenge in long-horizon agentic systems: how to make experience accumulation actually improve future behavior rather than merely serving as a reward signal or a static retrieval store. The co-evolution loop — where online RL rollouts continuously refine a structured skill library, which in turn conditions subsequent rollouts — is a concrete instantiation of the "skill discovery" problem that limits sample efficiency in tool-use agents. The hierarchical separation of execution primitives from workflow-level orchestration priors mirrors architectural distinctions in classical hierarchical reinforcement learning but operationalizes them through LLM-based distillation, making the approach directly applicable to modern MLLM-based agents. For researchers tracking LLM agent training methodology, SPyCE provides a practical alternative to pure GRPO/RLOO reward shaping that leverages trajectory structure rather than discarding it.