Skip to content

Digitizing Coaching Intelligence: An Agentic Framework for Holistic Athlete Profiling using VLM and RAG

🕒 Published (v1): 2026-06-26 19:53 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a six-agent LangGraph-orchestrated framework for automated athlete profiling that fuses MediaPipe-based kinematic tracking with VLM-based qualitative biomechanical reasoning. A 3×3 "Smart Grid" temporal chunking strategy reduces multimodal API overhead by 88%, while an LLM-as-a-Judge agent enforces cross-modal consistency before data is committed to a dual-persistence store. The system targets Sports Authority of India (SAI) mass recruitment scenarios where human coach scalability is the primary bottleneck.

Problem

Existing sports CV systems are quantitative counters (repetition volume) with no capacity for qualitative coaching judgement—fatigue onset, spinal articulation quality, compensatory movements. Manual coach assessment at national scale suffers from observer fatigue and subjectivity. No prior work closes the loop between geometric biomechanics and semantic physiological reasoning in a single automated pipeline.

Method

The framework consists of six specialized agents orchestrated by LangGraph in a directed acyclic graph with a feedback cycle:

  1. Agent 1 (Guardrail): Lightweight object-detection pre-screen; rejects corrupted/irrelevant video before any LLM token is consumed.
  2. Agent 2.1 (VLM): Processes 3×3 Smart Grid composites (9 frames at 2 FPS stitched into one image, left-to-right/top-to-bottom with green timestamp watermarks) using Llama-4-scout. Outputs a structured JSON with an Endurance Score (0–100) and timestamped qualitative flags (spinal articulation, core rigidity, grimacing).
  3. Agent 2.2 (CV): MediaPipe Pose extracts 33 3D landmarks. A Finite State Machine with hysteresis tracks rep cycles across UP/DOWN thresholds; joint angles computed via vector dot product \(\theta = \cos^{-1}\!\left(\frac{\vec{BA} \cdot \vec{BC}}{|\vec{BA}||\vec{BC}|}\right)\).
  4. Agent 3 (Assessor): Computes a composite score \(S_{\text{final}} = W_1 \cdot \text{Reps} + W_2 \cdot F_{\text{qual}} + W_3 \cdot E_{\text{vlm}}\) (default weights 0.4/0.4/0.2) and generates a natural-language feedback report via an LLM.
  5. Agent 4 (LLM-as-a-Judge): Cross-references CV and VLM outputs for logical contradiction. Confidence ≥ 70% → approve; < 70% → re-route to Agent 3 with correction instructions. Fail-closed: API timeout auto-assigns low confidence and flags for manual review.
  6. Agents 5 & 6 (Persistence): Dual-write to Google Sheets (structured tabular) and ChromaDB (semantic vector embeddings via gemini/embeddings-001). A RAG layer over ChromaDB supports natural-language coach queries.

Key Contributions

  • SAI-compliant multi-agent architecture decoupling quantitative (CV) and qualitative (VLM) analysis into parallel branches unified at a scoring node.
  • 3×3 Smart Grid temporal chunking: 9-frame composite images reduce API calls and token load by >88% while preserving temporal continuity for fatigue/form-degradation detection.
  • Multimodal LLM-as-a-Judge self-correction loop with structured tool-calling schema enforcement (confidence score, contradiction tags, severity level) and fail-closed routing.
  • Dual-persistence RAG pipeline: relational spreadsheet for tabular reporting + ChromaDB vector store enabling natural-language semantic athlete queries without SQL.
  • Weighted SAI-compliant scoring that penalizes biomechanical faults, preventing high-rep-poor-form athletes from outranking low-rep-good-form athletes.

Results

  • Smart Grid strategy reduces computational overhead by >88% compared to individual frame submission (stated claim; no ablation table provided).
  • Upstream CV baselines for context: Wang & Kong [6] achieve 99.5% counting accuracy at 30 FPS; Shi et al. [4] achieve 98.57% accuracy at 63.6 FPS—cited as prior art, not directly compared against the proposed system.
  • No head-to-head quantitative benchmarks of the full proposed framework are reported (e.g., no accuracy vs. ground-truth coach scores, no latency measurements, no hallucination rate before/after Judge).

Limitations

  • No empirical evaluation of end-to-end system accuracy against human expert ground truth; results are architectural claims only.
  • 88% efficiency reduction figure is asserted without a controlled experiment varying grid sizes or frame rates.
  • LLM-as-a-Judge confidence threshold (70%) is fixed by design choice with no sensitivity analysis.
  • Tested on push-ups and sit-ups only; generalizability to other SAI exercises is undemonstrated.
  • Dependence on external APIs (Llama-4-scout, Google Sheets, Gemini embeddings) introduces latency and cost variability not quantified.
  • Smart Grid assumes left-to-right temporal ordering is preserved by the VLM; no verification that the model respects this reading order across frame types.

Relevance to Harnesses / Meta-Harnesses

This paper is a domain-applied instance of a multi-agent meta-harness: LangGraph acts as the orchestration layer coordinating specialized sub-agents (guardrail, VLM, CV, assessor, judge, persistence) with conditional routing and retry loops—a structure isomorphic to generic agentic harness designs. The LLM-as-a-Judge feedback cycle is a concrete implementation of a self-correcting agent loop, relevant to harness designers studying how to enforce cross-modal consistency without human-in-the-loop. The dual-pipeline architecture (parallel ingestion branches merging at a synthesis node) and the fail-closed judge routing are reusable harness patterns applicable beyond sports analytics.