Skip to content

Evaluating Prompting Strategies with MedGemma for Medical Order Extraction

🕒 Published (v1): 2025-11-13 18:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper is a shared-task system description comparing three prompting strategies—1-shot, ReAct, and a simulated multi-agent agentic workflow—applied to MedGemma (4B and 27B) for structured medical order extraction from doctor-patient transcripts. Counterintuitively, the simplest 1-shot approach outperformed both complex frameworks, with MedGemma-27B 1-shot achieving an average score of 0.549 and placing 4th in MEDIQA-OE 2025.

Problem

Extracting structured medical orders (medications, labs, imaging, follow-ups) from conversational clinical transcripts is essential for automated EHR population, but requires handling context, negation, and order-reason relationships beyond simple NER. Prior work lacked systematic comparison of prompting complexity on this task, and the relative utility of agentic vs. direct prompting strategies for clean, annotated clinical data was unknown.

Method

All approaches use MedGemma (Google Gemma fine-tuned on medical corpora), tested at 4B and 27B scales:

  • 1-Shot Prompting: A single demonstration (conversation + structured JSON output) is prepended; the model directly generates a JSON array of orders with order_type, description, reason, and provenance (turn numbers).
  • ReAct Framework: Iterative Thought→Action→Observation cycles over segmented physician turns. Constraints are enforced during the Observation step (doctor-only orders, 20-word field limits, deduplication); inconsistencies trigger re-cycling. Post-processing normalizes and validates against the target JSON schema.
  • Agentic Workflow: Four sequential "agents" simulated within a single prompt context—Agent 1 (Identifier) detects candidate orders; Agent 2 (Mapper) pairs orders with reasons; Agent 3 (Structurer) produces JSON; Agent 4 (Validator) cross-checks against the transcript for hallucinations.

Output in all cases is a JSON array conforming to the same schema.

Key Contributions

  • Empirical comparison of 1-shot, ReAct, and simulated multi-agent prompting on a clinical IE task using a domain-specific open model (MedGemma).
  • Demonstration of "analytical over-processing": complex reasoning chains degrade precision on clean, manually annotated data by introducing spurious intermediate inferences.
  • Validation that scale (4B→27B) improves absolute scores while preserving the relative ranking of strategies.
  • Public release of prompts and code for all three approaches.

Results

MedGemma-4B on dev set: - 1-Shot: Avg 0.436 (Description Rouge1-F1: 0.516, Order Type Strict-F1: 0.602) - ReAct: Avg 0.277 - Agentic Workflow: Avg 0.111 (best provenance: 0.488, worst description: 0.09)

MedGemma-27B on test set: - 1-Shot: Avg 0.549 (Description Rouge1-F1: 0.591, Order Type Strict-F1: 0.703) — 4th place in MEDIQA-OE 2025 - ReAct: Avg 0.370 - MedGemma consistently outperformed base Gemma across all metrics (exact numbers not reported for base model comparison).

Limitations

  • Conclusions are dataset-specific: MEDIQA-OE uses clean, professionally annotated transcripts; complex frameworks may outperform 1-shot on noisy real-world ASR output.
  • The "agentic" workflow is simulated within a single model's context window, not a true multi-model system; independent-model agents could behave differently.
  • MedGemma-4B exhibited hallucinations in ReAct/Agentic reasoning steps and replicated few-shot examples rather than extracting novel orders.
  • Dataset limitations: sparse and inconsistent annotations, inter-annotator variability, lack of standardized clinical labels, and class imbalance across order types.
  • Agentic approach with 27B is computationally expensive; the 27B agentic condition is not reported.

Relevance to Harnesses / Meta-Harnesses

This paper constitutes a direct empirical test of a lightweight agentic harness pattern—sequential specialized sub-agents (Identifier → Mapper → Structurer → Validator) within a single prompt—against simpler prompting baselines on a structured extraction task. The finding that harness complexity can hurt performance on well-formed inputs (due to compounding intermediate errors across pipeline stages) is a concrete, quantified data point for harness designers calibrating when to use multi-step orchestration versus direct prompting. The ReAct loop's iterative constraint-validation mechanism is also a reusable harness motif worth noting. For researchers building meta-harnesses that select prompting strategies dynamically, this work provides evidence that data cleanliness and annotation structure should be features in the selection policy.