Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization

🕒 Published (v1): 2025-12-02 18:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a systematic evaluation framework for agentic code optimization applied to biomedical imaging "last mile" tool adaptation — generating pre/postprocessing functions that bridge domain gaps between production-level CV tools (Polaris, Cellpose, MedSAM) and bespoke lab datasets. A minimal Base Agent (LLM + execution feedback loop + data/API context) consistently outperforms hand-tuned expert baselines across all three tasks. Critically, adding architectural complexity (function banks, reasoning LLMs, AutoML, tree search) yields inconsistent and often negative results.

Problem

Adapting production-level biomedical CV tools to bespoke lab datasets requires weeks-to-months of expert manual coding; fine-tuning is impractical due to data scarcity (typically 10–100 labeled images available). Existing agentic approaches either target open-ended scientific discovery (too general, too complex) or build new ML pipelines from scratch (not applicable to adapting fixed pretrained tools). No systematic study existed of which agent design components actually matter for this narrow, high-value task.

Method

The framework wraps a pretrained production tool (unchanged) inside an iterative optimization loop:

1. Coding Agent (LLM) generates candidate pre/postprocessing function pairs given a Task Prompt, Data Prompt, and optional API list (98 curated OpenCV/Skimage/Scipy functions).
2. Execution Agent embeds functions into the pipeline, runs inference, and returns a validation score as feedback.
3. Per run: 20 iterations × 3 function pairs = 60 trials; 20 independent runs with different random seeds. Final score = max test score over top-15 validation-selected functions (to mitigate overfitting).

Base Agent = LLM + execution feedback + data context + API list. Four augmentations are ablated independently: LLM type (GPT-4.1, o3, Llama-3.3-70B), Expert Functions (in-context examples), Function Bank (top-3/bottom-3 prior functions retrieved each iteration), and AutoML Agent (periodic hyperparameter search over top-3 banked functions). Solution spaces are characterized along two axes: API Space (concentrated vs. dispersed, measured by edge-weight entropy of API co-occurrence graphs) and Parameter Space (easy vs. hard to optimize, measured by distribution gap between agent proposals and optimal range).

Key Contributions

• Demonstration that a simple iterative agent framework reliably exceeds expert-engineered (weeks–months of effort) baselines across three production biomedical imaging tasks.
• Systematic component-wise ablation revealing that common complex additions (function banks, reasoning LLMs, AutoML, tree search) have inconsistent, task-dependent effects rather than universal gains.
• A 2D solution-space characterization (API dispersion × parameter hardness) that explains the mixed effects and predicts which augmentations will help or hurt on a new task.
• Open-sourced framework; agent-generated functions merged into the official Polaris production codebase (anonymous PR).

Results

• Polaris (F1): Expert 0.841 → Base Agent 0.867 → Best (Add Expert Function) 0.929
• Cellpose (AP@IoU 0.5): Expert 0.402 → Base Agent 0.409 → Best (No API List) 0.417
• MedSAM (NSD+DSC): Expert 0.820 → Base Agent 0.971 → Best (No API List) 1.037
• Only the Small LLM (Llama-3.3-70B) fell below the expert baseline on Polaris (0.805) and MedSAM (0.918).
• AIDE tree-search agent (proprietary, complex) matched but did not exceed the Base Agent or Base Agent + Function Bank across all three tasks (Polaris: 0.872 vs. 0.867/0.889; MedSAM: 0.971 = tie).
• Non-agentic single-shot AutoML baseline performed worse than the Base Agent on all tasks.
• AutoML integration improved MedSAM (0.943→1.014) but degraded Polaris test scores (0.889→0.877) via validation overfitting; reducing AutoML frequency/iterations recovered test performance.

Limitations

• Evaluation covers only three tasks from one domain (biomedical imaging); generalizability to other scientific or software domains is unverified.
• Validation sets are very small (25–100 images), making overfitting a systemic risk, especially under aggressive hyperparameter search.
• The 2D solution-space taxonomy (API dispersion, parameter hardness) is characterized post-hoc from top-20 solutions; a priori characterization for new tasks requires additional analysis effort.
• Runtime is 1–2 days of compute per agent configuration; parallelism and cost tradeoffs are not fully analyzed.
• The study does not evaluate multi-step or compositional adaptation tasks where open-ended discovery agents might be more appropriate.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a code-optimization meta-harness: a framework that wraps an arbitrary black-box pipeline (the production tool) in an outer agent loop that iteratively proposes, executes, scores, and refines code components. The systematic ablation of harness design choices — what context to inject, whether to maintain a solution memory (Function Bank), when to invoke a sub-agent for hyperparameter search (AutoML Agent) — is precisely the kind of empirical meta-harness engineering study that informs how to compose agent loops for reliability. The finding that complexity is not monotonically beneficial and that the solution-space geometry determines which orchestration patterns help is a transferable design principle for anyone building harnesses that wrap fixed tools or APIs. The open-sourced framework is a concrete reusable scaffold for agentic code-optimization harnesses.