Skip to content

Toward Scalable Terminal Task Synthesis via Skill Graphs

🕒 Published (v1): 2026-04-28 14:53 UTC · Source: Arxiv · link

Why this paper was selected

Skill graphs enable scalable terminal task synthesis for training agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SkillSynth is an automated framework for synthesizing diverse terminal-agent training tasks by sampling compositional workflow paths from a large-scale scenario-mediated skill graph, then instantiating them into executable task instances via a multi-agent harness with dual verification and iterative repair. It produces 3,560 verified tasks in a single automated run at 95.7% oracle pass rate, and SFT on collected trajectories improves Qwen3-32B by up to 8.4 points on Terminal-Bench over non-graph baselines.

Problem

Training terminal agents (CLI-executing LLMs) is bottlenecked by the scarcity of high-quality, diverse execution trajectories. Existing synthesis approaches scale task count but do not control the joint diversity of scenarios traversed and skills exercised, leading to trajectory redundancy as characterized by overlapping (scenario, skill) pairs across task instances. Because the agent's learnable policy is confined to the empirical support \(\sum_{\sigma} p_D(\sigma|g) \sum_{\kappa} p_D(\kappa|\sigma,g) \log\pi(\kappa|\sigma,g)\), low-diversity data leaves large regions of the conditional product space \(\{(\sigma,\kappa) : \sigma \in \Omega, \kappa \in K_\sigma\}\) uncovered.

Method

Skill graph construction. Skills are harvested from ClawHub and public GitHub repos, then filtered for executability, determinism, and safety. An LLM infers pre- and postcondition scenarios for each skill; all scenarios are embedded, clustered with hierarchical agglomerative clustering (Louvain-based coarse bucketing), deduplicated, and cross-aligned via top-1,000 embedding-similarity retrieval + LLM compatibility judgment. The result is a directed multigraph \(G = (\Omega, K)\) with 82,073 scenarios and 57,214 skills connected by 185,529 LLM-verified bridges.

Inverse-frequency path sampling. Paths \(P = (\sigma_0, \kappa_1, \sigma_1, \ldots, \kappa_L, \sigma_L)\) of length \(L \in [1,7]\) are drawn with inverse-frequency weighting: source scenarios sampled \(\propto (\nu(\sigma)+1)^{-1}\), edges sampled \(\propto (\mu(\kappa)+1)^{-1}\). Monotone progression excludes revisited nodes/edges within a path; accepted paths update both counters, steering coverage toward the uniform over \(\Omega \times K\) (Algorithm 1).

Multi-agent synthesis harness. Each sampled path is passed to a two-agent pipeline: a Planner converts the path into structured sub-objectives; a tool-augmented Constructor generates all five task components (natural-language instruction, filesystem snapshot, Dockerfile, verification scripts, oracle solution). Dual verification checks solvability (execute oracle solution against verification scripts inside Docker) and rubric quality (LLM-as-Judge for test/instruction alignment and solution-hint leakage). Failures re-enter the constructor for up to \(R=3\) repair cycles of \(N_\text{tool}=20\) tool calls each; irrecoverable instances are discarded.

Key Contributions

  • Formal decomposition of execution trajectories as interleaved (scenario, skill) sequences, motivating a coverage criterion over the product space \(\Omega \times K\) as the training objective.
  • Scenario-mediated skill graph (\(G\) with 82K nodes, 57K edges, 185K bridges) as a scalable, community-extensible infrastructure; graph enumerates \(>16.6\)M paths of length \(\geq 7\).
  • End-to-end multi-agent harness (Planner + Constructor + verify-repair loop) producing 3,560 verified task instances from 3,721 sampled paths in a single automated run.
  • Demonstration that graph-guided synthesis yields harder and more diverse tasks than single-skill or randomly composed multi-skill baselines, with consistent downstream gains on Terminal-Bench 1.0 and 2.0 at multiple model scales.
  • SkillSynth task instances adopted in production to train Tencent Hy3 Preview.

Results

  • Harness yield: 92.0% pass both checks; 95.7% oracle pass rate; 721 tasks recovered by repair; average cost $27.3 per verified task instance.
  • Task difficulty: 38% unsolved (0/3 rollouts with Hy3 Preview), 37% in learnable range (1–2/3); Claude Opus 4.6 requires 37 steps on average and leaves 121 tasks unsolved after 3 rollouts.
  • Downstream SFT (Qwen3-32B + SkillSynth): 33.8 ± 3.1 on Terminal-Bench 1.0, 29.6 ± 1.6 on Terminal-Bench 2.0, outperforming Qwen 3 Coder 480B (23.9 ± 2.8) on TB 2.0 with 15× fewer parameters.
  • Ablation vs. baselines (Qwen3-32B): SkillSynth (+8.4 / +8.3) over single-skill, (+3.0 / +3.8) over multi-skill on TB 1.0 / TB 2.0.
  • Trajectory diversity: SkillSynth trajectories show 31% more unique scenario-skill pairs than single-skill and 19% more than multi-skill baselines.
  • Top proprietary baseline: Claude Opus 4.6 achieves 62.9 ± 2.7 on TB 2.0; GPT-5.3-Codex 64.7 ± 2.7.

Limitations

  • Repair effectiveness has a floor: corrupted filesystem snapshots from first-round LLM generation are rarely recovered across repair cycles, making re-sampling with higher temperature the only practical mitigation.
  • 77% of rubric failures stem from test/instruction misalignment (over- or under-specification), and these instances are excluded from RL training to avoid erroneous reward signals, reducing RL-usable data.
  • Skill graph quality depends on LLM-inferred scenarios and alignment judgments; manual review confirms each pipeline stage is necessary but the graph still contains noise without full human verification.
  • Evaluation restricted to Terminal-Bench 1.0/2.0 and Linux CLI; generalization to other OS environments or GUI-based terminal emulators is not demonstrated.
  • The paper's error analysis section is truncated in the provided text; partial implementation (42.2% of errors) is the dominant failure mode but full breakdown is unavailable.

Relevance to Harnesses / Meta-Harnesses

SkillSynth's multi-agent synthesis harness is a clear instance of a meta-harness: it does not execute tasks but manufactures the task instances and training trajectories that downstream agent harnesses consume. The planner–constructor–verifier–repairer loop is a structured multi-agent pipeline with iterative self-repair, mirroring patterns seen in code-generation meta-harnesses but applied to the problem of training-data synthesis at scale. The dual verification design (execution oracle + LLM rubric judge) operationalizes a quality gate that prevents harness-generated data from degrading RL reward signals, a concern directly relevant to anyone building automated curriculum or data-flywheel pipelines. The inverse-frequency path sampler provides a principled mechanism for harness-level diversity control, making SkillSynth a concrete reference for how graph-structured priors can replace ad-hoc prompt engineering in meta-harness design.