Skip to content

SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimuHome is a high-fidelity smart home simulator and 600-episode benchmark grounded in the Matter protocol that evaluates LLM agents on tasks requiring temporal reasoning, environmental feedback, and workflow scheduling. Unlike prior benchmarks that treat homes as static systems, SimuHome continuously propagates device actions to environmental variables (temperature, humidity, etc.) and supports time-accelerated evaluation of scheduled workflows. Evaluation of 18 agents reveals that workflow scheduling remains unsolved even with fine-tuning and reasoning-focused models.

Problem

Existing smart home benchmarks (HomeBench, Sasha, SAGE) model homes as static systems: they do not simulate how device operations continuously affect environmental variables over time, do not enforce operational dependencies between device commands (e.g., power on before mode change), and do not support time-based scheduling of workflows. Imitation learning on static datasets is also insufficient because a single user request can be satisfied by multiple valid action sequences that fixed annotations cannot cover.

Method

SimuHome divides simulation time into discrete 0.1-second ticks, deterministically computing the combined additive influence of all active devices on four environmental variables (temperature, illuminance, humidity, air quality) per tick. It implements 17 device types following the Matter protocol's cluster/attribute/command hierarchy, so device behavioral constraints (operational dependencies) mirror real hardware. Agents interact exclusively through APIs: exe_cmd, get_room_devices, schedule_workflow, and environment query tools—no direct state access.

The benchmark comprises 600 episodes across 12 categories (6 query types × feasible/infeasible): state inquiry (QT1), implicit intent inference (QT2), explicit device control (QT3), and three scheduling variants—time-based (QT4-1), event-driven (QT4-2), and coordinated (QT4-3). Episode generation: (1) randomized initial home state via dependency-aware device activation + time-acceleration; (2) structured goal + prerequisite action set (agents must call get_room_devices before acting); (3) GPT-5-mini query synthesis, validated by two annotators (Cohen's \(\kappa = 0.92\)). Feasible episodes with physical state changes use simulator-based evaluation; infeasible and QT1-feasible episodes use majority-vote LLM-as-a-Judge (\(\kappa = 0.826\) vs. human).

Key Contributions

  • SimuHome simulator: tick-based (\(\Delta t = 0.1\)s), fully reproducible smart home environment implementing the Matter protocol with continuous environmental variable propagation and time-accelerated workflow execution.
  • 600-episode benchmark: 12 evaluation categories covering state inquiry, implicit intent, explicit control, and three temporal scheduling variants, each with feasible and infeasible splits.
  • Dual evaluation protocol: simulator-based objective verification for state-change tasks; majority-vote LLM-as-a-Judge for natural-language response tasks.
  • Error taxonomy: 8 error types spanning environment perception, intent inference, device control, action planning, temporal reasoning, contradiction mishandling, and LLM-judge misclassification.
  • Empirical finding: workflow scheduling (QT4) is the persistent bottleneck across open-source, closed-source, fine-tuned, and reasoning-augmented models.

Results

  • Best overall performers: GPT-5.1 and Gemini-2.5-Pro top the leaderboard; GPT-5.1 achieves 100% on QT1-F and 94% on QT2-F.
  • Explicit device control (QT3-F): GPT-4.1 reaches 98%, GPT-5.1 100%, Gemini-2.5-Pro 94%—models handle this reliably.
  • Workflow scheduling (QT4-1/2/3): GPT-5.1 scores 92%/86%/60% on feasible QT4-1/2/3 respectively, substantially better than GPT-4.1 (84%/88%/50%) but at 3–5× the inference time (QT4-1-F: 121.1s vs. 26.6s for GPT-4.1).
  • Coordinated scheduling (QT4-3) is hardest: even GPT-5.1 reaches only 60% (feasible); most open-source models score ≤34%.
  • Fine-tuning (SFT) on successful trajectories: Gemma3-4B-it SFT improves QT1-F by 8pp and QT2-F by 8pp; Qwen3-32B SFT improves QT2-F by 22pp; workflow scheduling gains are marginal (≤10pp), confirming the bottleneck is in reasoning, not format.
  • Small open-source models (<7B): near-zero on all QT4 categories.
  • Infeasible detection: broadly weaker than feasible across all models; GPT-5.1 achieves 80%/94%/60% on QT4-1/2/3-IF.

Limitations

  • Environmental model is simplified (additive, per-room, 4 variables); cross-room and cross-device interactions are not modeled.
  • Only 17 device types; Matter protocol covers far more device classes.
  • Workflow commands that fail at scheduled execution time produce no error signal to the agent, creating evaluation ambiguity for partially correct plans.
  • Time-acceleration assumes the agent's reasoning time is negligible relative to simulated time, which may not hold for slow models in deployment.
  • LLM-as-a-Judge evaluation for infeasible episodes introduces non-determinism and potential bias despite majority voting.
  • Physical-to-simulation transfer is assumed but not empirically validated.

Relevance to Harnesses / Meta-Harnesses

SimuHome is directly relevant as a reference design for grounding agent harnesses in continuously-evolving, stateful environments — its core contribution is the harness architecture itself: a simulator that propagates device actions to environmental variables and feeds real-time feedback back to the agent loop, modeling exactly the kind of closed-loop evaluation scaffolding that meta-harnesses must implement. The finding that workflow scheduling remains unsolved across 18 agents, including reasoning-focused models and fine-tuned variants, is a direct signal for harness designers: temporal coordination (trigger conditions, deferred execution, interdependent workflows) cannot be delegated to the LLM alone and likely requires explicit scheduler components within the harness. The Matter-protocol grounding also makes this a practical blueprint for anyone building IoT or ambient-agent harnesses where device state is the environment substrate rather than text or code.