Skip to content

Knowledge-Informed Automatic Feature Extraction via Collaborative Large Language Model Agents

🕒 Published (v1): 2025-11-19 03:27 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Rogue One is a three-agent LLM framework for Automated Feature Extraction (AutoFE) on tabular data that replaces scalar accuracy feedback with qualitative assessments and integrates external knowledge via RAG. It operationalizes a "flooding-pruning" cycle: one agent floods candidate features, another prunes them statistically, and an orchestrating agent steers the search. Across 19 classification and 9 regression benchmarks it achieves MRR 0.76 and 0.91, substantially ahead of prior LLM-based methods.

Problem

Existing AutoFE methods treat the LLM as a monolithic generator guided only by scalar metrics (e.g., accuracy), rely entirely on parametric knowledge, and use rigid single-agent pipelines. These design choices limit exploration of the feature space, produce semantically opaque features, and provide no mechanism to integrate curated external domain knowledge.

Method

Rogue One formalizes feature extraction as finding the optimal transformation set \(H^* = \{h_1, \ldots, h_{m^*}\}\) that maximizes \(E(f^*(H(X_{\text{val}})), Y_{\text{val}})\) where \(f^*\) is an XGBoost/RF predictor. Three specialized agents operate in a closed iterative loop:

  1. Scientist Agent — reads cumulative test results from a shared Test Pool, identifies patterns, and produces a natural-language Focus Area directing the next extraction cycle; balances exploit vs. explore.
  2. Extractor Agent — translates the Focus Area into Python code expressing symbolic transformations \(h_i\); runs a discovery phase (df.head/df.describe) then an extraction phase producing ~17 candidate features per iteration stored in a Feature Pool.
  3. Tester Agent — evaluates all features via 5-fold XGBoost, then uses a Python environment and external knowledge to assess predictive power, stability, and redundancy, producing a rich markdown Feature Assessment and pruning low-value features.

All agents share a RAG tool backed by either a curated domain vector store (dense retrieval via Qwen3-Embedding-4B) or web search, decomposing queries into sub-queries before synthesis. The system runs for 10 iterations; the final feature matrix is the concatenation of all post-pruning pools: \(X^* = \{X_1^*; \ldots; X_{10}^*\}\).

Key Contributions

  • Rogue One: the first flexible, decentralized three-agent AutoFE framework with qualitative (not scalar) feedback.
  • Flooding-pruning strategy: high-volume generation (~17 features/iter) followed by statistically informed pruning to counter dimensionality growth.
  • RAG-augmented agents: external knowledge (domain literature or web) integrated into all agents rather than relying purely on parametric LLM knowledge.
  • Interpretability: every feature is accompanied by a semantic justification; qualitative Feature Assessments are human-readable markdown.
  • Scientific hypothesis generation: identified a candidate biomarker in the myocardial dataset.

Results

  • Classification (19 datasets, MRR): Rogue One 0.76 vs. OCTree 0.52 (2nd), LLM-FE 0.27, CAAFE 0.26, FeatLLM 0.28; wins on 12/19 datasets, sometimes by >5 pp accuracy.
  • Regression (9 datasets, normalized RMSE, MRR): Rogue One 0.91 vs. LLM-FE 0.56 (2nd), OpenFE 0.33; wins on 8/9 datasets.
  • Solution complexity (final feature count) shows no statistically significant Pearson correlation with \(\log(n)\) (ρ = 0.19, p > 0.05) or \(\log(p)\) (ρ = −0.24, p > 0.05), suggesting problem intrinsic complexity drives feature count.
  • Feature count range: 1 (balance-scale) to 92 (pc1) across datasets.

Limitations

  • No publicly available code; FELA excluded from comparison for the same reason, raising reproducibility concerns.
  • Experiments use a single LLM backbone (GPT-OSS-120B) without fine-tuning; cost and latency at 10 iterations × 3 agents × ~17 features are not reported.
  • Flooding-pruning is prompt-governed with no hard rules; Tester Agent behavior may vary non-deterministically.
  • RAG knowledge source selection is a manual design choice per domain, reducing automation.
  • No statistically significant correlation analysis means solution size is unpredictable; could produce very large feature sets on complex datasets.
  • Qualitative Feature Assessment is generated by an LLM and is not independently verified for correctness.

Relevance to Harnesses / Meta-Harnesses

Rogue One is a concrete production-grade meta-harness: it wraps multiple specialized sub-agents (Scientist, Extractor, Tester) into an orchestrated loop with shared persistent state (Test Pool, Feature Pool), qualitative feedback routing, and pluggable external knowledge. The flooding-pruning cycle is an explicit exploration-exploitation control mechanism analogous to the kind of adaptive loop management a meta-harness must provide. The use of RAG as a shared tool callable by any agent within the harness—rather than baked into a single model—exemplifies a clean harness-level resource injection pattern. For researchers building or studying harnesses, this paper demonstrates how role-specialization, structured inter-agent communication schemas (JSON Feature Explanations, markdown Assessments), and iterative state accumulation can be composed into a system whose quality exceeds any single monolithic agent.