Skip to content

SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; SimuHome: temporal- and environment-aware benchmark for smart home agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimuHome is a high-fidelity smart home simulator and 600-episode benchmark grounded in the Matter protocol that evaluates LLM agents on tasks requiring temporal reasoning, environmental feedback, and workflow scheduling. Unlike prior benchmarks that treat homes as static systems, SimuHome continuously propagates device actions to environmental variables (temperature, humidity, etc.) and supports time-accelerated evaluation of scheduled workflows. Evaluation of 18 agents reveals that workflow scheduling remains unsolved even with fine-tuning and reasoning-focused models.

Problem

Existing smart home benchmarks (HomeBench, Sasha, SAGE) model homes as static systems: they do not simulate how device operations continuously affect environmental variables over time, do not enforce operational dependencies between device commands (e.g., power on before mode change), and do not support time-based scheduling of workflows. Imitation learning on static datasets is also insufficient because a single user request can be satisfied by multiple valid action sequences that fixed annotations cannot cover.

Method

SimuHome divides simulation time into discrete 0.1-second ticks, deterministically computing the combined additive influence of all active devices on four environmental variables (temperature, illuminance, humidity, air quality) per tick. It implements 17 device types following the Matter protocol's cluster/attribute/command hierarchy, so device behavioral constraints (operational dependencies) mirror real hardware. Agents interact exclusively through APIs: exe_cmd, get_room_devices, schedule_workflow, and environment query tools—no direct state access.

The benchmark comprises 600 episodes across 12 categories (6 query types × feasible/infeasible): state inquiry (QT1), implicit intent inference (QT2), explicit device control (QT3), and three scheduling variants—time-based (QT4-1), event-driven (QT4-2), and coordinated (QT4-3). Episode generation: (1) randomized initial home state via dependency-aware device activation + time-acceleration; (2) structured goal + prerequisite action set (agents must call get_room_devices before acting); (3) GPT-5-mini query synthesis, validated by two annotators (Cohen's \(\kappa = 0.92\)). Feasible episodes with physical state changes use simulator-based evaluation; infeasible and QT1-feasible episodes use majority-vote LLM-as-a-Judge (\(\kappa = 0.826\) vs. human).

Key Contributions

  • SimuHome simulator: tick-based (\(\Delta t = 0.1\)s), fully reproducible smart home environment implementing the Matter protocol with continuous environmental variable propagation and time-accelerated workflow execution.
  • 600-episode benchmark: 12 evaluation categories covering state inquiry, implicit intent, explicit control, and three temporal scheduling variants, each with feasible and infeasible splits.
  • Dual evaluation protocol: simulator-based objective verification for state-change tasks; majority-vote LLM-as-a-Judge for natural-language response tasks.
  • Error taxonomy: 8 error types spanning environment perception, intent inference, device control, action planning, temporal reasoning, contradiction mishandling, and LLM-judge misclassification.
  • Empirical finding: workflow scheduling (QT4) is the persistent bottleneck across open-source, closed-source, fine-tuned, and reasoning-augmented models.

Results

  • Best overall performers: GPT-5.1 and Gemini-2.5-Pro top the leaderboard; GPT-5.1 achieves 100% on QT1-F and 94% on QT2-F.
  • Explicit device control (QT3-F): GPT-4.1 reaches 98%, GPT-5.1 100%, Gemini-2.5-Pro 94%—models handle this reliably.
  • Workflow scheduling (QT4-1/2/3): GPT-5.1 scores 92%/86%/60% on feasible QT4-1/2/3 respectively, substantially better than GPT-4.1 (84%/88%/50%) but at 3–5Ă— the inference time (QT4-1-F: 121.1s vs. 26.6s for GPT-4.1).
  • Coordinated scheduling (QT4-3) is hardest: even GPT-5.1 reaches only 60% (feasible); most open-source models score ≤34%.
  • Fine-tuning (SFT) on successful trajectories: Gemma3-4B-it SFT improves QT1-F by 8pp and QT2-F by 8pp; Qwen3-32B SFT improves QT2-F by 22pp; workflow scheduling gains are marginal (≤10pp), confirming the bottleneck is in reasoning, not format.
  • Small open-source models (<7B): near-zero on all QT4 categories.
  • Infeasible detection: broadly weaker than feasible across all models; GPT-5.1 achieves 80%/94%/60% on QT4-1/2/3-IF.

Limitations

  • Environmental model is simplified (additive, per-room, 4 variables); cross-room and cross-device interactions are not modeled.
  • Only 17 device types; Matter protocol covers far more device classes.
  • Workflow commands that fail at scheduled execution time produce no error signal to the agent, creating evaluation ambiguity for partially correct plans.
  • Time-acceleration assumes the agent's reasoning time is negligible relative to simulated time, which may not hold for slow models in deployment.
  • LLM-as-a-Judge evaluation for infeasible episodes introduces non-determinism and potential bias despite majority voting.
  • Physical-to-simulation transfer is assumed but not empirically validated.

Relevance to Agentic AI / LLM Agents

SimuHome directly advances the study of tool-use agents in long-horizon, temporally-extended settings—an open frontier for LLM agents where single-step tool invocation is insufficient. The benchmark's explicit separation of implicit intent inference, operational-dependency-aware device control, and multi-step temporal scheduling maps precisely onto the core capability gaps in current agentic pipelines. The finding that extended chain-of-thought (GPT-5.1) improves temporal reasoning at 3–5× latency cost raises a fundamental deployment tradeoff between reasoning depth and real-time responsiveness that is broadly relevant to any deployed LLM agent system. The time-accelerated simulation also demonstrates a viable pattern for building world-model environments where agents can pre-validate multi-step plans before committing to irreversible real-world actions—a safety mechanism applicable well beyond smart homes.