Skip to content

An Intelligent-Cloud Edge Multimodal Interaction System for Robots

🕒 Published (v1): 2026-07-16 07:39 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a cloud–edge framework for human–robot interaction that pairs a refined YOLO-based gesture detector (YOLO-DC) with coordinated LLM and VLM agents deployed in the cloud, while a TonyPi robot handles local sensing and motion control. YOLO-DC inserts CBAM attention into the YOLO11n neck and replaces the baseline regression loss with DIoU to improve detection of small or occluded gestures. System-level evaluation shows 95%/88%/82% success rates for single-action, composite-action, and vision-dependent tasks, respectively.

Problem

Lightweight on-device detectors degrade on small or occluded gestures in cluttered environments, and resource-constrained robots (e.g., Raspberry Pi–based platforms) cannot run large VLMs/LLMs locally. Existing systems also rely on single-modal, fixed-mapping command interfaces that lack semantic reasoning and multi-step task decomposition.

Method

YOLO-DC detector: CBAM is inserted after the C2PSA stage in the YOLO11n neck, adding ~0.1M parameters. Channel attention \(M_c\) and spatial attention \(M_s\) are applied sequentially: $\(F' = M_c(F) \otimes F, \quad F'' = M_s(F') \otimes F'\)$ The standard regression loss is replaced with DIoU: $\(\mathcal{L}_\text{DIoU} = 1 - \text{IoU} + \frac{\rho^2(b, b^{gt})}{c^2}\)$ which penalizes center-point distance even when boxes do not overlap.

Cloud–edge architecture: The cloud runs YOLO-DC for gesture detection, a VLM agent for open-vocabulary scene understanding (visual QA), and an LLM agent for intent parsing and action-sequence generation. A rule engine checks action compatibility and finite-state machines (FSMs) enforce execution order. All inter-component messages use a unified JSON schema carrying request type, recognized intent, object classes, positions, confidence scores, and action sequences. The TonyPi robot handles audio/image capture, JPEG compression (quality factor 85), transmission, text-to-speech feedback, and motion execution.

Key Contributions

  • YOLO-DC: YOLO11n augmented with CBAM in the neck and DIoU regression loss for improved small-gesture and occlusion robustness with minimal parameter overhead.
  • Dual-agent architecture: coordinated LLM + VLM agents with cross-modal fusion, rule-based validation, and FSM-based execution ordering.
  • Cloud–edge deployment pattern: structured JSON protocol decouples inference-heavy cloud reasoning from low-latency edge sensing/control on a TonyPi platform.

Results

  • Public gesture dataset (6 classes, 600 images): YOLO-DC precision 98.9%, mAP@0.5 90.7% vs. YOLO11n baseline 97.5% / 88.4%.
  • Custom dataset (3 classes): YOLO-DC precision 95.0%, mAP@0.5 92.7% vs. YOLO11n 79.0% / 86.7% (+16.0 pp precision, +6.0 pp mAP).
  • Ablation (public dataset): DIoU alone: mAP 89.6%; CBAM alone: mAP 88.9%; combined (YOLO-DC): mAP 90.7%, F1 96.9%.
  • Task success rates: single-action 95%, composite-action 88%, vision-dependent 82%.
  • Rule engine: rejects 90% of tested invalid action combinations.
  • VLM response latency: ~210 ms mean.
  • User study (30 participants, 5-point scale): mean satisfaction 3.69/5 across four interaction scenarios.
  • Response diversity (500 responses, 50 templates): repetition rate < 5%.

Limitations

  • Evaluated on small, controlled datasets (600 public images; custom dataset with 3 classes); generalization to unconstrained real-world gesture vocabularies is untested.
  • User study limited to 30 participants rating 4 predefined scenarios — insufficient for statistical significance or demographic generalization.
  • Cloud dependency introduces latency and requires network connectivity; no quantified end-to-end latency or failure-mode analysis under poor connectivity.
  • No production-grade security evaluation; authentication, encryption, and rate-limiting are noted as future work rather than implemented.
  • No comparison against cloud-based gesture detection alternatives or other multi-agent robotic systems at the system level.

Relevance to Agentic AI / LLM Agents

This paper is a concrete instantiation of a multi-agent robotic system where an LLM agent and a VLM agent operate as coordinated peers rather than monolithic models, with structured JSON as the inter-agent communication protocol and FSMs as the execution controller — a practical architecture pattern for grounded agentic systems. The rule engine + FSM layer provides a lightweight form of action verification and state management that complements the probabilistic outputs of foundation models, relevant to work on constrained or safe agentic planning. The cloud–edge split directly addresses the deployment gap for capable LLM/VLM agents on resource-limited hardware, a recurring challenge in embodied agent research. Compared to end-to-end approaches like RT-2, this system demonstrates how decomposing perception (YOLO-DC), reasoning (VLM/LLM), and control (FSM) into distinct, verifiable modules can yield interpretable, deployable agentic pipelines.