ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment¶

🕒 Published (v1): 2025-12-05 22:39 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ARCANE frames multi-agent alignment as a bilevel optimization problem where a manager agent learns to generate natural-language rubrics — weighted, verifiable criterion sets — that proxy stakeholder utility $U^*$ for downstream worker agents. The manager is trained via supervised fine-tuning followed by regularized Group Sequence Policy Optimization (GSPO), with penalties for excessive clarification and compute cost. Rubrics enable test-time steering and configurable trade-offs (e.g., correctness vs. conciseness) without retraining workers.

Problem¶

Existing rubric-based alignment methods treat rubric generation and rubric-conditioned optimization as separate processes; rubrics are assumed given rather than learned through stakeholder interaction. This limits adaptability to evolving or multi-objective preference contexts. Standard RLHF-style methods optimize against fixed training preferences and cannot maintain shared alignment across multiple interacting agents where stakeholder goals shift dynamically.

Method¶

ARCANE instantiates a three-role system: Stakeholder $S$ (latent utility $U^*$), Manager $M$ (rubric decomposer $D_\phi$), and Workers $W_1,\ldots,W_n$.

Rubric representation. A rubric $R = \{(c_j, w_j)\}_{j=1}^M$ pairs natural-language criteria with weights $w_j \in [0,1]$, $\sum_j w_j = 1$. Each criterion is scored by a verifier $\nu_j(c_j, x, y) \in [0,1]$, yielding a linearly additive proxy utility: $$\hat{u}_\phi(y \mid x) = \sum_{j=1}^M w_j \nu_j(c_j, x, y)$$

Stakeholder–manager dialogue. Before worker execution, the manager asks clarifying questions $q_{1:T}$ and synthesizes a rubric $R = D_\phi(x, q_{1:T}, a_{1:T})$ from the answers, subject to cost penalties for cognitive burden ($C_\text{clarify}$) and compute ($C_\text{compute}$).

Two-phase curriculum: - Stage I (SFT): Next-token prediction over synthetic manager–stakeholder dialogues and gold rubrics from GDPVal; initializes $\phi_0$. - Stage II (GSPO): For each task, $K$ rubrics are sampled, workers produce outputs, and scalar returns $r_k = U^*(y_k \mid x)$ are group-normalized into advantages $\hat{A}_k$. The sequence-level importance ratio is: $$s_k(\phi) = \left(\frac{\pi_\phi(R_k \mid x)}{\pi_\text{old}(R_k \mid x)}\right)^{1/|R_k|}$$ The objective clips $s_k \cdot \hat{A}_k$, adds a KL trust-region penalty, and subtracts $C_\text{clarify}$ and $C_\text{compute}$. Prioritized experience replay replays the bottom-$p$-th-percentile episodes by mean return to emphasize failures.

Test-time steering. Workers condition on $R^*$ as a prompt; rubric scores enable best-of-$K$ sampling, importance reweighting, or beam search without gradient updates.

Key Contributions¶

Formal bilevel framing of multi-agent alignment as learning an order-preserving proxy $\hat{u}_\phi \approx U^*$ via generated rubrics.
GSPO training objective with sequence-level (not token-level) importance ratios, KL regularization, and dual cost penalties for clarification and compute.
Prioritized experience replay at the rubric-group level to focus optimization on low-utility proposals.
Test-time configurability: stakeholders can directly inspect and edit $\{c_j, w_j\}$ without retraining.
Evaluation corpus of 219 labeled rubrics derived from GDPVal for multi-step reasoning and tool-use tasks.

Results¶

The paper text is truncated before the quantitative results section. The stated research questions to be answered are: - RQ1 (Usefulness): Whether learned rubrics increase worker output utility vs. unguided generation. - RQ2 (Faithfulness): Whether manager rankings at test time are consistent with stakeholder utility $U^*$. - RQ3 (Interpretability/Efficiency): Whether generated rubrics are compact and practically verifiable.

No benchmark numbers are present in the provided text.

Limitations¶

Evaluation relies on a single benchmark (GDPVal, 219 examples); generalization to other domains is unverified in the provided text.
The bilevel formulation assumes stakeholder preferences are complete and transitive (standard utility theory axioms), which may not hold in practice.
Workers are controlled only through prompting; the framework cannot modify worker parameters, limiting alignment depth when workers are opaque models.
$C_\text{clarify}$ difficulty classification relies on a fixed LLM prompt with hand-specified thresholds (easy/medium/hard), which may not transfer across domains.
The GSPO training requires a stakeholder simulator $U^*$, which must be constructed from gold rubrics — a form of human labeling not fully eliminated.

Relevance to Harnesses / Meta-Harnesses¶

ARCANE is directly a learned meta-harness: the manager agent acts as a dynamic orchestration layer that generates the evaluation and control criteria ($R^*$) used by worker agents, mirroring the role a harness plays in defining execution contracts. The two-phase curriculum (SFT warm-start → RL fine-tuning with GSPO) is a concrete template for training a harness controller end-to-end rather than hand-authoring prompt templates or reward functions. The rubric-as-proxy-utility abstraction generalizes beyond ARCANE: any harness that routes tasks, scores outputs, and feeds signals back to workers can be framed as a rubric decomposer operating on stakeholder utility. The cost-penalty terms ($C_\text{clarify}$, $C_\text{compute}$) are a principled approach to the orchestration efficiency problem common to all meta-harnesses.