Skip to content

MOSAIC: Modular Orchestration for Structured Agentic Intelligence and Composition

🕒 Published (v1): 2026-05-30 12:31 UTC · Source: Arxiv · link

Why this paper was selected

MOSAIC: modular orchestration substrate for structured agentic composition

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MOSAIC is a structured agentic framework for automated data science that decomposes model selection into a staged, memory-grounded pipeline: task profiling → case/code retrieval → blueprint construction → LLM-mediated synthesis → RL-guided refinement. It addresses the lack of structured workflow representation in existing LLM-based data-science agents, making modelling decisions verifiable and reusable. Evaluated on financial time-series tasks, it outperforms AutoML and agentic baselines on predictive, distributional, and financial metrics.

Problem

Existing data-science agents (DS-Agent, ResearchAgent, TS-Agent) treat LLMs as unconstrained code generators: they lack structured intermediate representations of the workflow being built, persistent memory of prior modelling decisions, and principled long-horizon refinement strategies. AutoML systems (AutoGluon, Optuna) have the complementary weakness—fixed search spaces that cannot reason over task descriptions or synthesize novel architectures from reusable components. Neither class of system produces decisions that are traceable, composable, or systematically reusable.

Method

MOSAIC decomposes model construction into five staged substeps:

  1. Task profiling — semantic EDA produces \(\phi_{task} = (\phi_{schema}, \phi_{stat}, \phi_{visual}, \phi_{feat})\), multimodally augmented by a VLM for visual time-series patterns.
  2. Case + code retrieval — \(\phi_{task}\) queries a case bank \(\mathcal{E}_{case}\) of prior task–solution pairs and a code repository \(\mathcal{E}_{code}\) of annotated, reusable modules (embeddings, encoders, forecasting heads, training routines).
  3. Blueprint construction — an intermediate representation \(B\) is synthesized specifying selected modules, composition order, input–output shape contracts, and execution constraints; LLM generation is conditioned on \(B\) rather than producing free-form code.
  4. Repository-grounded model generation — architecture-family analysis of the top-\(k\) retrieved candidates drives module extraction and recombination, producing a new candidate \(m_{k+1}\); all \(k+1\) candidates are executed and evaluated in parallel (warm-up phase).
  5. RL-guided long-horizon refinement — a learned offline RL policy (Implicit Q-Learning) selects structured high-level refinement actions \(a_t \in \mathcal{A}_{ref}\) (e.g., add normalisation, change LR schedule, modify dropout); a frozen LLM executor translates each action into concrete code edits. The policy–executor factorisation is: $\(p(s_{t+1} | s_t) = \sum_{a_t, e_t} p(s_{t+1} | e_t, s_t)\, p(e_t | a_t, s_t)\, \pi(a_t | s_t)\)$ Offline training uses failure-aware trajectory branching: hard failures (syntax errors, NaN loss, timeout) and soft failures (sustained validation degradation) trigger rollbacks to the last valid checkpoint; failed branches become negative supervision and populate per-state invalid-action masks \(\mathcal{I}_{invalid}(s)\), preventing repeated bad edits. At deployment, the policy queries \(Q(s_t, a)\) over \(\mathcal{A}_{valid}(s_t)\).

Key Contributions

  • Formulation of automated data science as memory-grounded modular workflow construction over a structured solution space \(\mathcal{A}(\mathcal{T}, \mathcal{K})\).
  • Blueprint-based, repository-grounded synthesis pipeline: case retrieval → module extraction + semantic annotation → blueprint \(B\) → LLM-conditioned code generation.
  • Failure-aware offline RL for long-horizon code refinement: trajectory branching, rollback, and invalid-action masking convert execution failures into reusable supervision for an IQL policy.
  • Demonstration on financial time-series (forecasting and generation) with joint evaluation of predictive accuracy, distributional fidelity, execution reliability, and downstream financial risk metrics (\(\Delta\)VaR, \(\Delta\)ES, \(\Delta\)Sharpe).

Results

  • Forecasting (RMSE vs. TS-Agent): −8% on LOB, −5% on Crypto, −3% on Stock (averaged across GPT-4o, GPT-5.4, Claude Opus 4, Nova Pro backbones); single-backbone peak gains of −19% (LOB-Claude) and −8% (Crypto-GPT-5.4).
  • Forecasting vs. AutoGluon: average RMSE reduction of 33% on LOB.
  • Risk metrics on Crypto vs. TS-Agent: average \(\Delta\)Sharpe reduced by >21%; GPT-4o backbone alone shows 46% reduction.
  • Ranking radar charts show MOSAIC forming the outermost contour on nearly all axes across forecasting and generation tasks.
  • Execution success rate and decision traceability improve relative to all agentic baselines (precise numbers are in figures/appendices not fully reproduced in the text excerpt).

Limitations

  • Evaluated only on financial time-series (forecasting and generation); generalizability to other data science domains (computer vision, NLP, tabular) is undemonstrated.
  • Knowledge banks (\(\mathcal{E}_{case}\), \(\mathcal{E}_{code}\), \(\mathcal{E}_{refine}\)) must be pre-populated with domain-relevant cases and modules; cold-start quality and maintenance burden are not analyzed.
  • The RL policy is trained offline on trajectories generated for financial time-series; transfer to new task types would require re-collection of training trajectories.
  • Blueprint construction and module extraction rely on LLM calls whose correctness and shape-contract adherence are not formally verified beyond execution-based validation.
  • Dependency on proprietary LLM backends (GPT-5.4, GPT-4o, Claude Opus 4) introduces cost and API availability constraints; no open-weight ablation is reported.

Relevance to Harnesses / Meta-Harnesses

MOSAIC is a direct instance of a meta-harness for automated data science: it orchestrates a heterogeneous pipeline of retrieval, LLM synthesis, execution, and learned policy components through an explicit intermediate representation (the blueprint), which plays exactly the role a meta-harness's workflow specification plays in general agentic frameworks. The blueprint-as-IR pattern — where the harness constructs a structured plan and then delegates execution to a sub-executor (here, a frozen LLM) — is a generalizable architectural principle relevant to anyone building orchestration layers over LLM agents. The failure-aware trajectory branching and invalid-action masking for offline RL refinement offer a concrete, reusable technique for meta-harnesses that must learn to improve their own execution policies from observed failures without destroying valid intermediate state.