Towards Continuous Intelligence Growth: Self-Training, Continual Learning, and Dual-Scale Memory in SuperIntelliAgent

🕒 Published (v1): 2025-11-28 18:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SuperIntelliAgent pairs a trainable small diffusion model with a frozen LLM verifier to enable annotation-free continual self-improvement. The learner generates images, the verifier decomposes prompts into semantic conditions and evaluates alignment via chain-of-thought, producing chosen/rejected pairs for DPO fine-tuning. A dual-scale memory—short-term in-context traces within a refinement thread, long-term LoRA parameter updates across threads—allows lifelong intelligence growth with no human supervision.

Problem

Generative vision models are trained once on fixed datasets and cannot adapt post-deployment; obtaining labeled preference data for every new domain is prohibitively expensive. Without continual adaptation, deployed models drift from real-world data distributions. Existing self-critique approaches do not close the loop into persistent parameter updates in an annotation-free, online manner.

Method

Learner–verifier pairing. A trainable diffusion model \(L_\theta\) and a frozen vLLM verifier \(V\) interact in iterative threads. Given prompt \(p\), \(V\) decomposes it into semantic conditions \(C(p) = \{c_1,\ldots,c_n\}\) via chain-of-thought. For each generated image \(x_t\), \(V\) computes per-condition confidence \(s_t^i = V_\text{eval}(c_i, x_t) \in [0,1]\). If \(\min_i s_t^i \geq \tau\), the thread succeeds; otherwise \(V\) emits critique \(f_t = V_\text{critique}(C(p), s_t)\) and the learner regenerates \(x_{t+1} = L_\theta(p, f_t)\).

Auto-DPO pair construction. Successful trajectories yield preference pairs \(\mathcal{D}_\text{DPO} = \{(p, x_k^-, x^+)\}\) where all prior iterates are negatives. Pairs with score margin below 0.15 are discarded.

Diffusion-DPO loss. Since exact likelihoods are intractable, the DPO objective is approximated via denoising loss: $\(\mathcal{L}_\text{DDPO}(\theta) = \mathbb{E}_{(p,x^-,x^+)}\!\left[\mathcal{L}_\text{denoise}(\theta; p, x^+) - \mathcal{L}_\text{denoise}(\theta; p, x^-)\right]\)$

Dual-scale memory. Short-term memory preserves verifier feedback traces within a thread (up to \(T=5\) steps). Long-term memory consolidates reasoning patterns into \(L_\theta\)'s parameters via LoRA (\(\Delta W = AB^\top\), \(r \ll \min(d,k)\)) applied asynchronously from a replay buffer \(\mathcal{B}_\text{replay}\) that retains only trajectories showing measurable progress.

Asynchronous pipeline. Inference and training run in parallel threads; a bounded staleness constraint (\(\leq K\) steps) maintains stability between the generating and updating model.

Key Contributions

• Annotation-free Auto-DPO: every inference thread is converted into self-supervised preference data without human labels.
• Dual-scale memory architecture separating in-context short-term traces from LoRA-based long-term parameter consolidation.
• Progress-gated replay buffer that enforces curriculum ordering and prevents catastrophic forgetting.
• Infrastructure-agnostic design compatible with AutoGen, Semantic Kernel, and similar frameworks.
• Empirical demonstration that continual learning on hard cases only (sparse DPO pairs) yields consistent benchmark gains.

Results

• GenEval: Janus-1.3B improves 58.41% → 69.62% (+11.21 pp); Janus-Pro-7B improves 76.31% → 83.54% (+7.23 pp). Counting accuracy nearly doubles for Janus-1.3B (23.75% → 46.25%). Two-object composition: 60.61% → 84.85% (1.3B), 82.83% → 92.93% (7B).
• DPG-Bench: Janus-1.3B: 83.09% → 84.57% (+1.48); Janus-Pro-7B: 87.13% → 88.35% (+1.22).
• T2I-CompBench: Janus-1.3B: 52.43% → 54.70% (+2.27); Janus-Pro-7B: 60.61% → 62.09% (+1.48).
• SD v1.5 + SuperIntelliAgent (overall GenEval 0.525) surpasses SD v2.1 (0.17) and baseline SD v1.5 (0.06) despite using the weaker backbone.
• Training efficiency: on DPG-Bench (1,065 prompts), only 241 DPO pairs were generated; most prompts were skipped as already meeting threshold.

Limitations

• Verifier is GPT-4o-mini / o1-mini (commercial, closed-source); inference costs and API dependence are not analyzed.
• Gains on T2I-CompBench are modest, suggesting spatial/compositional generalization remains difficult.
• The replay buffer is small relative to model size; individual batches may be sampled multiple times, risking overfitting to recent data.
• Evaluation is restricted to text-to-image tasks; generalization of the framework to other modalities is asserted but not demonstrated.
• No ablation isolating the contribution of dual-scale memory versus DPO data quality or replay buffer strategy.

Relevance to Harnesses / Meta-Harnesses

SuperIntelliAgent is a self-improving harness architecture: it wraps a generative model inside a verifier-driven orchestration loop that continuously generates, evaluates, and trains on its own outputs—the harness itself is the training curriculum. The asynchronous inference–training pipeline and infrastructure-agnostic adapter interface represent a concrete instantiation of a meta-harness pattern, where the scaffolding layer (judge + improver + replay buffer) drives both runtime behavior and model evolution. The dual-scale memory design—separating orchestration-level context (short-term) from parameter-level consolidation (long-term)—is directly relevant to work on harnesses that need to accumulate state across sessions without rearchitecting the base model. For researchers building harnesses, this paper demonstrates how a frozen judge component and a progress-gated replay buffer can replace hand-curated training data, making the harness itself the data flywheel.