ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment¶

🕒 Published (v1): 2025-12-05 22:39 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ARCANE frames LLM alignment as a multi-agent bilevel optimization problem in which a manager agent learns to elicit stakeholder preferences via dialogue and synthesize them into weighted, natural-language rubrics that guide worker agents at test time. The manager is trained via SFT on synthetic dialogues then fine-tuned with a sequence-level policy gradient method (GSPO) that jointly penalizes poor utility, excessive stakeholder interaction, and verification cost. Learned rubrics enable configurable, retraining-free steering and outperform unguided best-of-N baselines on a challenging file-manipulation benchmark.

Problem¶

Training-time alignment methods (RLHF, DPO) optimize fixed preferences and cannot adapt when stakeholder goals shift at deployment. Existing test-time reward models (GenRM, GRAM) provide opaque scalar judgments without exposing which criteria drive evaluation or how to reconfigure them without retraining. In multi-agent deployments where preferences are distributed and dynamic, neither paradigm offers interpretable, auditable, and negotiable alignment signals.

Method¶

ARCANE introduces three interacting roles: a Stakeholder with a latent utility function U, a Manager that elicits and encodes preferences, and Workers that execute tasks. The manager's core output is a rubric* R = {(c_j, w_j)}: a finite set of weighted natural-language criteria paired with rule-based or model-based verifiers, forming a linearly additive proxy utility û_ϕ.

Training is a two-phase curriculum: 1. SFT warm-start: a large reasoning model generates synthetic stakeholder dialogues and reference rubrics; the manager decomposer is fine-tuned via next-token prediction over dialogue turns and rubric text. 2. GSPO reinforcement phase: K rubric proposals are sampled per task, each conditions a worker to produce an output, and the stakeholder utility r_k = U*(y_k|x) is the scalar return. Advantages are group-normalized. The GSPO objective uses a length-normalized sequence-level importance ratio (replacing GRPO's token-level ratio) with KL regularization to a reference policy, plus explicit penalty terms for clarification cost C_clarify (logarithmic in number/difficulty of questions) and compute cost C_compute (logarithmic in LLM verification latency/cost). Prioritized experience replay upsamples low-return episodes.

Test-time steering: workers receive R* as a natural-language prompt; candidates can be ranked via û_ϕ for best-of-N selection, importance reweighting, or beam search without any gradient update.

Key Contributions¶

Formalization of multi-agent alignment as a bilevel optimization with partial observability of U*, with proof that any strictly proper supervision operator (pointwise, pairwise, listwise, rubric) identifies the same ordinal utility structure.
ARCANE framework: manager-as-rubric-decomposer with structured stakeholder Q&A, separating preference elicitation from worker execution.
GSPO training objective with sequence-level importance ratios, cost regularizers for interaction burden and verification overhead, and prioritized experience replay at the rubric-group level.
Configurable test-time alignment: rubrics can be directly inspected, edited, or overridden by stakeholders without retraining workers.
Empirical evaluation on 219 GDPVal tasks requiring multi-step file manipulation and tool use.

Results¶

Usefulness (N=1): mean return 0.58 (No Rubric) → 0.59 (SFT) → 0.62 (GSPO) → 0.70 (Oracle).
Usefulness (N=8): 0.58 → 0.68 → 0.74 → 0.81; GSPO vs. SFT difference statistically significant (Wilcoxon p=0.0182, mean per-task improvement +0.044).
Faithfulness (NDCG@8): 0.7998 (base, no dialogue) → 0.8103 (SFT) → 0.8722 (GSPO); GSPO yields +8.3% Precision@3 and reduces mean rank swaps among top candidates (2.20 → 2.13).
Domain breakdown: GSPO gains strongest in content/communication (+11.5%, NDCG=0.905) and legal/compliance (+12.5%, 0.848); operational tasks regress (–8.1%, 0.830); data-analysis flat (+1.9%, 0.822).
Interpretability: GSPO rubrics match Oracle density (~12 criteria, 17–18 tokens/criterion), allocate +14% more weight to quality-stage criteria, and show lower weight variance (3.5 vs 4.2) than SFT.

Limitations¶

Experiments validate only manager-plus-single-worker; multi-worker coordination dynamics are unverified.
GDPVal tasks are discrete episodes; long-horizon, extended-deployment evaluation is absent.
No direct comparison to alternative test-time methods (GenRM, GRAM), so interpretability/performance tradeoffs are not characterized.
No structural regularizer prevents the manager from learning spurious criteria that correlate with but are not causally responsible for utility; causal or invariance penalties are left to future work.
Evaluation set is small (44 tasks); bootstrap confidence intervals are wide at N=1.

Relevance to Agentic AI / LLM Agents¶

ARCANE directly addresses a core open problem in long-horizon agentic systems: how to keep a delegation hierarchy (manager→workers) aligned with evolving, incompletely specified stakeholder preferences without retraining the worker layer. The rubric-as-coordination-artifact idea is broadly applicable to any multi-agent scaffold where agents cannot share parameters (closed-source APIs, heterogeneous systems), making it practically relevant to frameworks like AutoGen and CAMEL. The GSPO formulation, which penalizes both interaction cost and verification overhead, provides a concrete template for building economically rational alignment agents. The test-time configurability—stakeholders can edit rubric criteria and weights directly—bridges LLM alignment research with the human-in-the-loop control requirements that real agentic deployments demand.