Towards Trustworthy Multi-Turn LLM Agents via Behavioral Guidance¶

🕒 Published (v1): 2025-12-12 10:03 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a three-component framework—task profiler, reasoning module, and generation module—that embeds LLM agents in an explicit RL-style action–observation–reward loop to achieve verifiable and reliable multi-turn behavior. The task profiler acts as a meta-learner that routes strategy selection; the reasoning module builds a persistent "Rule Bank" of observation→action mappings; the generation module enforces constraint compliance via validation or deterministic code synthesis. Initial experiments on Guess My Number and Wordle show consistent improvement over ICL-augmented baselines.

Problem¶

Multi-turn LLM agents (ReAct, Reflexion, AutoGPT, etc.) treat trust as an implicit outcome of competence: reasoning is embedded in opaque text-generation loops with no mechanism to verify that internal logic aligns with task constraints or environment feedback. Existing approaches lack persistent, inspectable reasoning state and post-hoc constraint enforcement that adapts to structurally diverse task environments.

Method¶

The framework wraps any LLM in an RL-formalized task environment interface (observation space O, action space A, reward signal R, validity check, transition function). Three co-evolving components operate across epochs:

Task Profiler: A lightweight LLM-based function that analyzes environment variables and classifies task structure (temporal dependency: sequential vs. cumulative; constraint intensity: light vs. heavy). It prescribes which reasoning and generation strategies to use, and is re-run each epoch to detect dynamic shifts.
Reasoning Module: After each epoch, analyzes successful trajectories at profiler-specified temporal granularity and extracts "if [observation condition] → then [action]" rules stored in a persistent Rule Bank with success rates and usage history. Rules generalize across epochs via testing/filtering on new trajectories.
Generation Module: At each turn, validates the proposed action against the Rule Bank and environment constraints via fvalidity. On failure, falls back to deterministic enumeration of all valid candidates filtered by accumulated constraints (implemented as generated code for constraint-heavy tasks like Wordle).

Baseline comparisons: (a) RL-prompt-only agent without ICL, (b) same with randomly sampled successful ICL trajectories.

Key Contributions¶

A modular behavioral guidance framework that operationalizes trust as verifiability (inspectable reasoning mappings) + reliability (constraint-compliant outputs) within the agent's own feedback loop.
A task profiler grounded in cognitive science and RL principles that performs adaptive strategy selection without solving the task directly.
A Rule Bank mechanism providing persistent, auditable procedural memory that accumulates and generalizes observation→action mappings across epochs.
Separation of reasoning and generation layers, enabling deterministic fallback synthesis (code generation) for constraint-heavy environments.
Evaluation metrics for multi-turn trustworthiness: reasoning consistency ratio and constraint-compliance ratio.

Results¶

All experiments use GPT-4.1-mini, 30 epochs × 20 trajectories, reported with 95% CIs.

Guess My Number: Guided agent achieves steady reward improvement and stabilizes at higher average reward vs. both baseline variants (with/without ICL), which show no consistent improvement over epochs. After epoch 15, rules stabilize and reasoning consistency ratio increases monotonically (with exploration dips at epochs 8, 11, 13). Guided agent exhibits narrower reward confidence intervals indicating lower trajectory variance.
Wordle: Baseline agents (with/without ICL) show no consistent improvement and frequently violate constraints despite correctly restating them in natural language. After activating code-based generation at epoch 10, the guided agent achieves >60% immediate recovery of initially invalid outputs via deterministic fallback. Guided agent completes Wordle in fewer turns and at higher success rates than baselines. Compliance ratios remain consistently high with narrow variance.

Limitations¶

Evaluated only on two simple game environments (Guess My Number, Wordle); generalization to open-ended or real-world agentic tasks is undemonstrated.
Rule Bank details (registration, filtering, deduplication across runs) are omitted from the paper; full mechanism requires inspecting the codebase.
Task profiler is a "minimal instantiation"—an LLM prompted as a cognitive strategy engine with fixed categories; no learned or data-driven profiling.
All experiments use GPT-4.1-mini; performance with larger or reasoning-specialized models is left to future work.
Deterministic enumeration fallback may not scale to high-cardinality action spaces beyond structured games.
No ablation isolating the task profiler's contribution from the reasoning and generation modules independently.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses a central gap in agentic LLM research: the lack of intrinsic verifiability and constraint reliability in multi-turn interaction loops. By formalizing the agent's environment in RL terms and externalizing reasoning into an auditable Rule Bank, it provides a blueprint for agents whose behavior can be inspected and steered—critical for deploying agents in industrial or safety-sensitive settings. The separation of task profiling, reasoning, and generation into modular, co-evolving components aligns with broader architectural trends (HRM, brain-inspired modular agents) but operationalizes them as a learnable system rather than a static hierarchy. The emphasis on constraint-compliant generation via programmatic synthesis is particularly relevant as agentic systems increasingly operate over structured action spaces (APIs, GUIs, databases).