Skip to content

Fine-Tuned Multi-Agent Framework for Detecting OCEAN in Life Narratives

🕒 Published (v1): 2026-07-13 23:26 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A 15-agent harness (three Mistral-7B sub-agents per OCEAN trait, each fine-tuned via MLM and LoRA to a HIGH, LOW, or NEUTRAL trait orientation) feeds binary decisions, extracted evidence, and confidence scores into a Qwen judge that produces final three-class personality predictions from long-form life narrative interviews. The framework improves average macro-F1 by approximately 8% over the best single-agent baseline and demonstrates that heterogeneous model assignment—small models for localized evidence extraction, a larger-context model for aggregation—outperforms homogeneous multi-agent configurations.

Problem

Single-model LLM personality assessment suffers from latent pretraining biases that cause inconsistent trait inferences and fails to capture distributed, subtly expressed cues across long narratives. Prior encoder baselines (BERT/RoBERTa) are limited to short texts and coarse labels, while prompting-based LLM approaches lack psychometric grounding and interpretability.

Method

Architecture. Fifteen Mistral-7B-Instruct sub-agents are instantiated—one per (trait \(t\), level \(i\)) pair where \(t \in \{O,C,E,A,N\}\) and \(i \in \{\text{High, Neutral, Low}\}\). Each sub-agent performs binary classification: does the narrative exhibit its assigned trait level?

MLM fine-tuning. Sub-agents are fine-tuned with a self-supervised masked language modeling objective on trait-stratified narratives. Masking is emotion-aware: 15% of tokens are emotional words (drawn from SenticNet, filtered to 2,914 words with primary emotion score \(\geq 1.0\)) and 15% are non-emotional, totaling 30% masking per sequence.

LoRA adaptation. Each sub-agent applies LoRA (\(r=4\), \(\alpha=8\), dropout 0.05) to the \(q,k,v,o\) projection layers, yielding ~4.2M trainable parameters. The merged weight update is \(\theta_{\text{merged}} = \theta_{\text{base}} + \lambda\tau\) where task vector \(\tau = \frac{\alpha}{r}BA\) and \(\lambda=0.3\).

IPIP-NEO grounding. Sub-agent prompts include trait-level IPIP-NEO facet keys: positively keyed items \(F_t^H\) for HIGH agents, negatively keyed \(F_t^L\) for LOW agents, and a semantically blended \(F_t^N\) for NEUTRAL agents. Each sub-agent also receives a role preamble specifying its trait orientation.

Judge aggregation. A Qwen judge \(J_t\) receives the full transcript plus structured outputs \(\{b_t^{(i)}, e_t^{(i)}, c_t^{(i)}\}_{i=1}^3\) from three sub-agents for trait \(t\), along with facet definitions, and produces the final label \(\hat{y}_t \in \{\text{High, Neutral, Low}\}\).

Data. 1,535 life narrative interview transcripts from the SPAN dataset; continuous ground-truth scores tercile-binned to produce class labels. 50% used for MLM fine-tuning, 50% for evaluation.

Key Contributions

  • Multi-agent framework inducing psychometrically grounded HIGH/NEUTRAL/LOW trait orientations in LLM sub-agents through combined MLM fine-tuning, LoRA adaptation, role prompts, and IPIP-NEO facet keys.
  • Judge-mediated evidence aggregation that reconciles conflicting binary signals across the three sub-agents per trait into a single multi-class prediction.
  • Demonstration that heterogeneous model assignment (Mistral for extraction, Qwen for aggregation) outperforms homogeneous multi-agent configurations due to differing context-window requirements.
  • Directional semantic alignment metric (cosine margin \(\delta_t^i = s_H - s_L\) for HIGH agents, inverted for LOW) quantifying whether sub-agent evidence aligns with intended trait orientation.
  • PCA visualization of LoRA task vectors confirming trait-specific and level-specific weight-space separation.

Results

  • Full framework macro-F1 (avg 0.416): O=0.424, C=0.366, E=0.416, A=0.415, N=0.458; best single-agent (Qwen CoT) achieves avg ~0.383, representing ~8% absolute improvement.
  • Baselines: RoBERTa avg ~0.255; Mistral CoT avg ~0.249; all-Mistral multi-agent avg ~0.353; all-Qwen multi-agent avg ~0.357; Mistral sub-agents + Qwen judge (proposed) is best across most traits.
  • Ablation F1 drops (avg): removing IPIP-NEO keys −15.6%, removing MLM −11.3%, removing trait-level cues −10.6%, removing sub-agents −7.9%.
  • Per-class accuracy: HIGH and LOW are substantially stronger than NEUTRAL across all traits (e.g., Neuroticism: HIGH=48.8%, LOW=47.6%, NEUTRAL=41.2%); NEUTRAL is consistently hardest.
  • MLM fine-tuning: baseline (no fine-tuning) achieves ~9–10% test accuracy and 12–14% Acc@5; fine-tuned sub-agents reach avg 52.7% test accuracy and 80.2% Acc@5, with perplexity reduced from ~8.x to ~6.5–7.5.
  • Semantic alignment: full model achieves highest directional margin on all 10 agent/trait pairs; omitting IPIP-NEO keys produces negative margins on several agents (e.g., N_high: −0.060).

Limitations

  • NEUTRAL class accuracy is poor across all traits, reflecting the fundamental difficulty of moderate trait expression; the system has no mechanism specifically targeting this class boundary.
  • Inference is sequential across 15 sub-agents with GPU memory cleared between runs, making latency proportional to agent count.
  • Evaluated on a single dataset (SPAN life narratives); generalizability to other text genres (social media, essays) is untested.
  • Conscientiousness consistently underperforms relative to single-agent baselines under multi-agent decomposition, suggesting the architecture is not uniformly beneficial.
  • Tercile binning of continuous scores introduces label-boundary artifacts and discards ordinal information within bins.
  • Only two backbone models (Mistral-7B, Qwen) explored; the finding that judge model size matters for aggregation is not systematically characterized.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instantiation of a typed multi-agent harness: specialized sub-agents perform bounded, single-responsibility inference tasks and pass structured outputs (binary decision + evidence + confidence) to an orchestrating judge that performs synthesis and final-stage reasoning. The split between extraction agents and aggregation judge mirrors the decomposition pattern central to meta-harness design, and the finding that heterogeneous model assignment—small models for local extraction, larger-context model for aggregation—outperforms homogeneous ensembles is an empirical result about agent role specialization directly relevant to harness architecture. The ablation study further quantifies the contribution of individual harness components (MLM adaptation, role conditioning, psychometric grounding, sub-agent decomposition), providing rare controlled evidence on which structural elements of a multi-agent pipeline drive performance gains. The sequential execution model with memory clearing between sub-agents also highlights a practical resource-management pattern in constrained-GPU harness deployment.