Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment

🕒 Published (v1): 2025-12-05 06:46 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes Collective Agency (CA), an open-ended alignment value defined by four entangled pillars (Knowledge, Power, Vitality, Benevolence), and Dynamic Alignment, a self-improving GRPO-based framework that lets an LLM iteratively align itself to CA without human-labeled data. Applied to a 20B model, the approach produces a model strongly preferred on CA alignment while preserving general NLP benchmark performance.

Problem

Static alignment objectives (HHH: helpfulness, honesty, harmlessness) are vulnerable to reward hacking, risk epistemic capture by compressing diverse values into a single channel, and become difficult to scale via RLHF as models grow. Existing AI-feedback self-improvement methods (Constitutional AI, Self-Rewarding LMs) remain tethered to these conventional, narrow objectives and do not target open-ended agentic growth.

Method

Collective Agency (CA) is defined as the infinite expansion of agency across spacetime, operationalized through four mutually reinforcing pillars: Knowledge, Power, Vitality, and Benevolence. A single holistic CA score is assigned per response (not per pillar) to prevent isolated optimization.

Dynamic Alignment runs in two phases: 1. Training data generation: o4-mini generates 1,000 task prompts via an iterative multi-agent pipeline—goal generation → prompt generation → LLM evaluator feedback → revision loop—producing open-ended prompts that elicit CA without mentioning it explicitly. 2. Self-improving phase: The policy model (gpt-oss-20b) generates G=8 candidate responses per prompt (with a CA-oriented system prompt), scores each via its own self-reward mechanism (0–5 integer rubric), computes GRPO advantages, and updates parameters—excluding the system prompt from gradient computation to prevent prompt overfitting. Training runs until reward convergence on a single H100 NVL (94 GB).

Hyperparameters: batch size 32, lr=5×10⁻⁶, ε=0.2, β=0.04, top-p=1.0, T=1.0 for generation; greedy decoding for self-reward scoring.

Key Contributions

• Definition of Collective Agency as an open-ended, non-static alignment value structured around four entangled agentic pillars.
• Dynamic Alignment framework combining automated prompt generation and GRPO-based self-rewarding, requiring no human-labeled preference data.
• Demonstration that a self-rewarding loop can align a 20B model to an abstract, holistic value without degrading general NLP capabilities.
• A 1,000-prompt CA evaluation dataset and multi-agent generation pipeline for open-ended alignment assessment.

Results

• CA alignment (win rate vs. base gpt-oss-20b): CA-aligned model wins 87.2 ± 4.1% vs. base 12.8 ± 4.1% (100-prompt held-out set; GPT-4.1 judge, positional-bias-controlled).
• IFEval (instruction-following): base 73.0 ± 0.75% → CA-aligned 72.4 ± 1.08% (statistically equivalent, 95% CI, 5% margin).
• GPQA Diamond (PhD-level science): base 37.0 ± 2.97% → CA-aligned 36.6 ± 1.61% (statistically equivalent).
• AIME 2025 (math reasoning): base 34.4 ± 3.85% → CA-aligned 34.4 ± 6.94% (statistically equivalent, 10% margin).

Limitations

• Evaluated on a single model family (gpt-oss-20b); no comparison against alternative self-improving baselines (CAI, Self-Rewarding LMs).
• Self-rewarding loop risks value drift and evaluation circularity—the model judges its own outputs with no external anchor.
• CA is inherently abstract; no granular per-pillar scoring is used or validated, making it hard to diagnose which aspects improve.
• Evaluation relies on a single LLM judge (GPT-4.1); no human evaluation or multi-model judging.
• No analysis of how CA alignment affects standard safety objectives (harmlessness, honesty); potential regression is uncharacterized.
• Dataset generation uses o4-mini (external dependency), not the policy model itself.
• Single GPU training; scalability to larger models not demonstrated.

Relevance to Agentic AI / LLM Agents

Dynamic Alignment directly targets the challenge of building agents that can self-improve toward open-ended agentic objectives—perceiving, planning, acting, and renewing—without human oversight bottlenecks, which is central to scalable autonomous agent development. The self-rewarding GRPO loop is a concrete mechanism for agents to bootstrap alignment from their own outputs, relevant to any system where human feedback is impractical at inference scale. The CA value framework explicitly encodes the full agentic cycle (perception → planning → action → renewal) as alignment targets, connecting to ongoing debates about how to specify objectives for increasingly autonomous systems. The planned extension to multi-agent negotiation for oversight further situates this work within the collective/multi-agent alignment subfield.