Skip to content

ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions

๐Ÿ•’ Published (v1): 2025-01-01 ยท Source: NeurIPS ยท Venue: NeurIPS 2025 ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

ContextAgent is the first framework for context-aware proactive LLM agents that continuously ingests wearable sensor streams (egocentric video, audio, smartphone notifications) to autonomously predict when a user needs help and execute tool chains โ€” without any explicit user instruction. The authors also introduce ContextAgentBench, a 1,000-sample benchmark spanning 9 daily-life scenarios and 20 tools to evaluate this capability.

Problem

Existing LLM agents are reactive: they require explicit user instructions to act. Prior proactive agents either operate in closed desktop environments (computer screenshots, keyboard inputs) or use static rule-based triggers, and none integrate external tool calling for actual service delivery. This leaves a gap for an agent that can perceive open-world, egocentric sensor context, personalize to user history, and invoke arbitrary tools proactively.

Method

ContextAgent operates in two stages:

  1. Proactive-oriented context extraction: VLMs prompted with in-context learning (not zero-shot, to bias descriptions toward cues relevant for proactivity) convert egocentric video to visual context C_V; a speech recognition model produces audio context C_A; smartphone notifications N are appended directly. Persona context P (preferences, identity, historical behaviors) is derived from historical data via LLM extraction.

  2. Context-aware reasoner: An LLM fine-tuned with LoRA (rank 8, AdamW lr=1e-4, cosine schedule, 5 epochs on 8ร—A6000) using SFT + chain-of-thought traces distilled from Claude-3.7-Sonnet. At inference, the model receives (C, P), emits explicit <think> traces, a proactive score P_S โˆˆ {1โ€“5}, and an ordered tool chain T_C = {(t_i, a_i)}. Proactive services are triggered only when P_S โ‰ฅ ฮธ (user-adjustable); tools are called sequentially and their results integrated into the final response.

ContextAgentBench was seeded with 200 human-annotated exemplars then scaled to 1,000 via an automated LLM diversification pipeline (scenario-aware and proactive-score-aware generation with human+script verification). ContextAgentBench-Lite (300 samples) pairs entries with real egocentric video and audio scraped from Pexels or self-collected.

Key Contributions

  • First formalization of the context-aware proactive agent task: (T, P_S, T_C, R) = A(S, P) driven by open-world wearable sensors.
  • ContextAgent framework with ICL-guided proactive-oriented VLM context extraction and a CoT-distilled, LoRA fine-tuned reasoner with think-before-act capability.
  • ContextAgentBench: 1,000 samples, 9 scenarios, 20 tools, proactive score labels, and annotated tool chains; ContextAgentBench-Lite: 300 samples with paired raw sensor data.
  • Demonstrated that an 8B/7B-scale fine-tuned model can match or exceed 70B-scale and proprietary LLM baselines on proactive prediction and tool calling.

Results

  • ContextAgentBench (Llama-3.1-8B-Ins base): ContextAgent achieves Acc-P = 0.874, F1 = 0.626, Acc-Args = 0.448 โ€” improvements of +8.5% Acc-P, +7.0% F1, +6.0% Acc-Args over the strongest baseline (Vanilla SFT).
  • ContextAgentBench-Lite (Qwen2.5-7B base): +6.2% Acc-P, +3.0% F1, +7.6% Acc-Args over best baseline; matches or exceeds ICL baselines using GPT-4o and Llama-70B.
  • OOD generalization (held-out 3 scenarios): ContextAgent (DeepSeek-R1-7B) achieves 90.9% Acc-P, 68.9% F1, 51.6% Acc-Args; outperforms best baseline by 8.3% Acc-P, 10.7% F1, 1.9% Acc-Args.
  • Vs. proprietary LLMs (OOD): ContextAgent (7B) reaches 89.3% Acc-P vs. GPT-4o 88.6%, GPT-o3 86.8%; Claude Sonnet 4 leads at 91.3% Acc-P but ContextAgent surpasses it on Acc-Args (48.9% vs. 48.0%).
  • Ablation โ€” missing modalities: Removing vision drops Acc-P by 17.9% and F1 by 23.3%; removing audio causes smaller but significant degradation.
  • Ablation โ€” personas: Removing personas decreases Acc-P by up to 9.0%, F1 by 12.3%, Acc-Args by 12.6%.

Limitations

  • Persona context is sourced from the benchmark's static pool rather than continuously updated from live sensor history in real deployments.
  • ContextAgentBench-Lite (300 raw-sensor samples) shows performance degradation for all methods relative to the text-description variant, indicating a gap between idealized textual context and real sensor processing.
  • Tool chain accuracy degrades at longer chain lengths (the conservative prompting strategy inflates missed detection rates for complex multi-tool scenarios).
  • The benchmark remains simulated/semi-synthetic for most samples; real-world longitudinal evaluation across users is absent.
  • Privacy implications of continuous egocentric sensing are not addressed.

Relevance to Agentic AI / LLM Agents

ContextAgent advances the frontier from reactive, instruction-following agents to proactive, sensor-driven agents โ€” a key unsolved problem for personal AI companions and embodied systems. The think-before-act SFT pipeline (distilling reasoning traces from a frontier model into a 7โ€“8B model) is a directly reusable recipe for practitioners building lightweight proactive agents. The introduction of ContextAgentBench fills a benchmark gap that previously forced evaluations into closed desktop environments, providing the community with a standardized test bed for tool-augmented proactive behavior. This work is directly relevant to anyone tracking wearable AI, multimodal agents, or the shift from reactive to autonomous agentic systems.