Skip to content

Operation Veja: Fixing Fundamental Concepts Missing from Modern Roleplaying Training Paradigms

🕒 Published (v1): 2025-12-12 22:14 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current roleplaying LLMs fail to simulate believable characters because their training paradigms model surface personality traits rather than the causal internal logic of a character's mind. The paper proposes VEJA (Values, Experiences, Judgments, Abilities), a data curation framework that structures character profiles around four interconnected psychological pillars. A small pilot study shows VEJA-guided human-written data is strongly preferred over state-of-the-art synthetic baselines by an LLM judge.

Problem

Existing roleplaying training paradigms—RAG, fact-based priming, literature-based generation, and synthetic distillation—cannot model the deliberative, value-conflicted reasoning that makes characters believable. RAG cannot enumerate combinatorially explosive human value systems; fact-based priming strips values of contextual interplay; literary dialogue implies but does not expose internal deliberation as training signal; synthetic generation hits a recursive quality ceiling because the generator itself lacks the capacity to balance conflicting values.

Method

VEJA is a conceptual framework for structuring character profiles along four causally linked axes: - Values: High-level motivators driving goals ("I value ambition/productivity"). - Experiences: Past events that ground Values and Judgments in evidentiary context. - Judgments: Concrete opinions and heuristics produced by Values filtered through Experiences. - Abilities: Skills/knowledge domains that determine vocabulary, interests, and expressive range.

The causal chain is: Experiences → Values → Judgments → expressed via Abilities. For a pilot study, 15 human writers were given a 10-day story timeline and a VEJA profile for the character Makise Kurisu (Steins;Gate) and wrote conversations grounded in that profile. This was compared against a synthetic baseline where Gemini Pro 2.5 generated equivalent conversations from the same timeline without a VEJA profile. Evaluation used 100 blind pairwise LLM-as-judge comparisons with Gemini 2.5 Flash.

Key Contributions

  • Taxonomic critique of four dominant roleplaying training paradigms and their specific failure modes.
  • The VEJA framework: a four-pillar character definition schema grounded in principles from dramatic arts (Stanislavski).
  • A pilot dataset of VEJA-guided human-curated dialogues for Makise Kurisu as a proof-of-concept "gold standard."
  • Empirical evidence (LLM-as-judge, N=100) that VEJA-guided human curation significantly outperforms synthetic generation.

Results

  • Out of 100 pairwise comparisons: VEJA-curated dialogues preferred 43 times, synthetic baseline preferred 28 times, 29 ties.
  • Excluding ties: VEJA win rate ≈ 60.6% vs. synthetic 39.4%.
  • Judge reasoning consistently cited superior narrative continuity, nuanced reactions, and tighter personality-dialogue linkage in VEJA outputs; synthetic outputs flagged as generic, out-of-character, or over-indexed on single traits.

Limitations

  • Pilot study uses a single character (Makise Kurisu) from a single source (Steins;Gate); generalization is unproven.
  • Dataset is small-scale; no sample sizes reported for the datasets themselves.
  • Manual curation is labor-intensive and not scalable without future semi-automated tooling.
  • Human-vs-machine confound: preferred outputs are human-written, so it is unclear how much improvement comes from VEJA vs. human writing quality per se.
  • No ablation studies on individual VEJA components due to time constraints.
  • Evaluation relies on LLM-as-judge (Gemini 2.5 Flash), which carries known biases; no human evaluation conducted.

Relevance to Agentic AI / LLM Agents

This paper directly targets the character-consistency and internal-state modeling problem in roleplaying agents, which are a specific class of LLM agents that must maintain persistent personas across multi-turn interactions. The VEJA framework is a data curation schema—not a new architecture—making it relevant to anyone building agent training pipelines where character coherence and value-consistent reasoning are required. The critique of synthetic data generation's recursive quality ceiling is particularly salient for agentic self-improvement loops: if the base model cannot model conflicting values, it cannot generate data that teaches value-conflict resolution. The work also underscores that "gold standard" human-curated data, structured by domain-specific ontologies, may be a necessary complement to synthetic scaling for achieving genuine deliberative agency.