Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions

🕒 Published (v1): 2026-07-19 23:06 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM agents are increasingly used to simulate human behavior under disaster/disruption scenarios, but individually plausible agent reasoning often fails to reproduce empirical population-level activity distributions. This paper shows that embedding demographic profiles (ACS), time-use routines (ATUS), and urban spatial constraints into agent initialization, memory, and decision prompts substantially improves statistical realism, validated against an independent heatwave survey.

Problem

Ungrounded LLM agents generate narratively coherent but statistically unrealistic population behavior: need-based reasoning produces internally consistent individual actions that fail to reproduce empirical activity distributions, temporal patterns, and heterogeneous group-level adaptation. This validity gap is especially dangerous for disaster planning, where simulated outputs inform shelter demand estimates, resource allocation, and intervention design.

Method

The authors modify the AgentSociety multi-agent simulation framework by injecting empirical context at three levels:

1. Agent initialization: Each agent receives a demographic profile drawn from ACS (stratified by employment status, work mode, disability/household constraints) and a Markov-chain-derived daily activity schedule from ATUS at hourly resolution.
2. Agent memory and prompts: The empirical time-varying activity probability \(P(a_t)\) for the current hour is inserted directly into the LLM decision prompt, instructing the agent to follow baseline routines unless environmental context (e.g., heat) warrants deviation.
3. Spatial constraint layer: OpenStreetMap + SafeGraph POI data for central Philadelphia (bounding box 39.90°N–40.00°N, −75.24°W to −75.09°W) constrains feasible activity destinations.

The testbed is the July 2024 Philadelphia heatwave. Two agent systems are compared: (a) ungrounded AgentSociety baseline (need-based reasoning only) and (b) the grounded framework. Validation uses two independent benchmarks: ATUS-derived normal-day profiles and an independent household survey (\(n=603\)) conducted during the actual heatwave, reserved entirely as a holdout.

Key Contributions

• Demonstration that empirical grounding (demographic profiles + time-use routines + spatial constraints) substantially improves statistical realism of LLM-agent population simulations, not just narrative plausibility.
• A concrete pipeline for injecting ATUS Markov-chain activity sequences into LLM agent memory and prompts as soft behavioral priors rather than hard rules.
• External validation methodology using an independent survey benchmark withheld from all training, calibration, and prompting stages.
• Quantification of the "plausibility gap": ungrounded agents produce coherent-sounding behavior while reproducing only 20.6% of observed heatwave response amplitude; grounded agents reach 46.4%.
• Identification of remaining gaps (work schedule adjustments, reduced nonessential travel) as evidence that additional empirically grounded adaptation modules—not stronger prompting—are needed.

Results

Normal-day baseline reconstruction (vs. ATUS empirical profiles): - Mean correlation: 0.528 → 0.912 (grounded) - Mean squared error: 0.066 → 0.008 (grounded)

Heatwave-day activity profiles (vs. independent household survey, \(n=603\)): - Mean correlation: 0.349 → 0.836 (grounded) - Mean squared error: 0.098 → 0.012 (grounded)

Heatwave behavioral adaptation amplitude (heatwave-minus-normal change magnitude): - Empirical benchmark mean change magnitude: 0.08156 - Ungrounded baseline: 0.01628 (20.6% of empirical amplitude) - Grounded model: 0.03772 (46.4% of empirical amplitude, >2× improvement) - Grounded model correctly reproduced dominant decrease in biological needs and increase in leisure; underestimated work/study and travel adaptation.

Limitations

• Heatwave benchmark relies on self-reported 24-hour recall, subject to recall error and subjective interpretation.
• Survey used broader employment categories than the simulation model; work-mode-specific adaptation (hybrid vs. in-person) could not be validated at fine granularity.
• Validation targets group-level activity distributions, not individual trajectories, activity sequences, or destination-level mobility.
• Missing behavioral mechanisms: indoor thermal conditions, household energy constraints, building thermal performance, perceived risk, and work flexibility are not modeled—these likely explain the remaining 53.6% of unrecaptured adaptation amplitude.
• Single city (Philadelphia) and single hazard type (heatwave); generalizability untested.

Relevance to Harnesses / Meta-Harnesses

This paper is directly relevant as a blueprint for LLM agent harness design: it shows how a simulation harness can embed structured empirical data (Markov-chain schedules, ACS profiles) into agent initialization, memory, and decision-prompt pipelines to constrain LLM generativity without replacing it. The modified AgentSociety framework functions as a meta-harness—a wrapper that augments base LLM agent scaffolding with externally grounded behavioral priors. The two-stage validation architecture (held-out benchmark withheld from all harness inputs) also models a rigorous evaluation harness pattern applicable to any agent simulation system. For researchers building harnesses for human-in-the-loop or population-level simulation, this work establishes empirical grounding as a necessary harness layer, not an optional post-hoc step.