Skip to content

SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents

🕒 Published (v1): 2025-09-29 04:54 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimuHome is a high-fidelity, time-accelerated smart home simulator and 600-episode benchmark for evaluating LLM agents on device control and workflow scheduling tasks. It grounds simulation in the Matter industry protocol and introduces dual evaluation via simulator state-checking and LLM-as-a-Judge. Across 18 evaluated agents, workflow scheduling is the hardest unsolved category, even after SFT and alternative frameworks.

Problem

Existing smart home benchmarks treat the home as a static system: they neither model how device operations continuously affect environmental variables (temperature, humidity, illuminance, air quality) nor enforce operational dependencies between commands nor support time-accelerated evaluation of scheduling tasks. Imitation learning on static datasets is also insufficient because a single user request can be fulfilled by multiple valid action sequences that fixed annotations cannot cover.

Method

SimuHome simulates a configurable home with 17 device types and 4 per-room environmental variables. Time is discretized into 0.1-second ticks; state transitions are fully deterministic given a seed. Devices affect environmental variables additively each tick via parameterized update equations. Agents interact only through a defined API (Matter protocol commands: exe_cmd, get_room_devices, schedule_workflow, etc.) and cannot observe global state directly.

The benchmark spans 6 query types (QT1: state inquiry; QT2: implicit intent; QT3: explicit control; QT4-1/4-2/4-3: time-based, event-driven, and coordinated scheduling), each with feasible and infeasible variants, yielding 12 categories × 50 episodes = 600 total. Episodes are constructed via a three-step pipeline: (1) seeded randomization of initial home state respecting device dependency order, (2) structured goal and prerequisite-action generation, (3) GPT-5-mini query synthesis followed by graduate-student validation (\(\kappa = 0.92\)).

Evaluation uses two complementary methods: simulator-based (direct final-state comparison against structured goal for QT2/3/4 feasible) and LLM-as-a-Judge with majority vote over 3 queries (\(\kappa = 0.826\) vs. human) for all infeasible episodes and QT1-feasible.

Key Contributions

  • SimuHome simulator: tick-based, deterministic, time-acceleratable, grounded in Matter protocol with continuous environmental feedback
  • 600-episode manually validated benchmark spanning 12 evaluation categories including infeasible variants
  • Dual automated evaluation pipeline (simulator state + LLM-as-a-Judge)
  • Empirical evaluation of 18 agents (small open-source to GPT-5.1) with SFT and alternative framework ablations
  • Error taxonomy: 5 feasible error types (EP, II, DC, AP, TR) + 3 infeasible types (CM, CB, LJ)

Results

  • GPT-5.1 achieves the highest scores: QT1-F/IF 100/94%, QT2-F/IF 100/80%, QT3-F/IF 94/78%, QT4-1-F/IF 92/60%, QT4-2-F/IF 72/44%, QT4-3-F/IF 50/34%
  • Gemini-2.5-Pro: QT1-F/IF 96/78%, QT2-F/IF 94/60%, QT3-F/IF 76/46%, QT4-1-F/IF 92/72%
  • GPT-4.1 (non-reasoning): QT3-F 98%, QT4-1-F only 50%
  • Coordinated scheduling (QT4-3) remains hardest: even GPT-5.1 achieves only 50% feasible, 34% infeasible
  • GPT-5.1 requires 3–5× the inference time of GPT-4.1 (e.g., QT4-2-F: 135.1s vs. 28.7s), making it impractical for real-time deployment
  • SFT on successful trajectories (Gemma3-4B, Qwen3-32B) yields modest gains but does not resolve workflow scheduling
  • Small open-source models (<7B) largely fail: Llama3.2-1B achieves 0% on all scheduling tasks

Limitations

  • Only 4 environmental variables and 17 device types; cross-room and cross-device interaction effects are not modeled
  • Infeasible variants cover only 3 constraint categories (non-existent device, physical limit, temporal contradiction); real-world edge cases may be broader
  • Evaluation of infeasible episodes relies on LLM-as-a-Judge, introducing model bias despite \(\kappa = 0.826\) human agreement
  • Sim-to-real transfer is assumed but not empirically validated against physical Matter deployments
  • Temporal coordination failures are attributed to model reasoning capability without ablations that isolate the role of context length vs. arithmetic precision

Relevance to Harnesses / Meta-Harnesses

SimuHome is itself an evaluation harness that wraps a deterministic simulator, a structured episode generator, and a dual-method evaluation pipeline into a reproducible benchmark infrastructure — a pattern directly analogous to meta-harnesses that orchestrate agents against environments with automated scoring. The three-step episode-generation pipeline (state construction → goal/prerequisite synthesis → query synthesis + human validation) is a reusable harness authoring pattern for grounding LLM-agent benchmarks in physical semantics. The time-acceleration mechanism is a concrete technique for making long-horizon harnesses tractable at evaluation scale, reducing multi-minute real-world waits to immediate simulator ticks. For researchers building meta-harnesses that pre-validate agentic plans before real-world execution, SimuHome's proposal to use its simulator as a world-model sandbox for action pre-validation is the most directly relevant design contribution.