End-to-End Automated Logging via Multi-Agent Framework

🕒 Published (v1): 2025-11-23 16:45 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoLogger is a hybrid framework that solves the complete end-to-end software logging pipeline via a two-stage design: a fine-tuned classifier (Judger) decides whether to log, then a multi-agent system (Locator + Generator) resolves where and what to log. On three large-scale Java codebases, it achieves 96.63% F1 on the whether-to-log decision and improves end-to-end logging quality by 16.13% over the strongest baseline.

Problem

Existing automated logging tools skip the whether-to-log decision, assuming it has already been made, which forces them into roles as pure code generators that simultaneously worsen overlogging and miss underlogging. Additionally, monolithic LLM approaches attempt control-flow analysis, data-flow tracing, and stylistic generation in a single static forward pass, causing hallucination of variable names, misplaced log insertions, and style violations.

Method

AutoLogger decouples logging into two sequential stages:

Stage I — Judger: A Qwen2.5-Coder-14B model fine-tuned via LoRA on a binary classification task (LOG / NO_LOG). Training positive samples are constructed by removing one log from a method that has \(n \geq 1\) logs, simulating an underlogged state. BM25 retrieves a similar method as a one-shot in-context example during training/inference to incorporate project-specific conventions.

Stage II — Multi-Agent System (MAS): Activated only for LOG-classified methods. Two ReAct-style agents operate sequentially under a stateful procedural Orchestrator (not an LLM): - Locator: Iteratively reasons over method code using tools (backward slicing, BM25 similar-case retrieval) to output the source code annotated with <<need_logging>> markers indicating insertion points. - Generator: Receives the annotated code from the Orchestrator, then uses variable_extractor (AST-based static scope analysis via tree-sitter) and backward_slicing to identify available variables and control/data dependencies, then retrieves stylistically similar logging examples to produce the final log statement.

The Orchestrator parses agent outputs via regex, reformats tool results into structured LLM-friendly context, and enforces global timeouts and retry logic for robustness.

Key Contributions

• First systematic treatment of the whether-to-log decision as a first-class sub-task in automated logging.
• AutoLogger hybrid architecture: fine-tuned classifier gating a specialized MAS, avoiding wasted LLM compute on logging-saturated methods.
• Program-analysis tool pool (backward slicing, AST-based variable extraction, BM25 retrieval) that grounds agent reasoning in verifiable code facts.
• Demonstrated backbone-agnostic generalizability: the framework improves multiple underlying LLMs consistently.

Results

• Whether-to-log (Judger): 96.63% F1-score, substantially outperforming general-purpose LLM baselines on this decision.
• End-to-end quality: 16.13% improvement in LLM-as-a-judge score over the strongest baseline across Doris, Kafka, and ZooKeeper (160k–1.09M LOC; 1.9k–3.4k logging statements per project).
• Framework is generalizable: consistently boosts performance of various backbone LLMs (specific backbone ablation results not reproduced in the provided text).

Limitations

• Evaluation is Java-only; generalizability to other languages is unverified.
• The Judger requires per-project fine-tuning data, limiting applicability to codebases with insufficient logging history.
• Backward slicing is static; dynamic or inter-procedural dependencies are not captured.
• BM25 retrieval for style is shallow — does not account for semantic similarity beyond token overlap.
• End-to-end evaluation relies on LLM-as-a-judge, which inherits biases of the judge model.

Relevance to Harnesses / Meta-Harnesses

AutoLogger is a concrete domain-specific meta-harness: a stateful Orchestrator coordinates heterogeneous components (a fine-tuned classifier, two ReAct agents, and a static analysis tool pool) through a defined pipeline with timeout enforcement, retry logic, and structured inter-agent context transfer — all canonical meta-harness concerns. The gating pattern (classifier → expensive MAS only on positive cases) is a resource-aware orchestration strategy directly applicable to general harness design. The Orchestrator's role of parsing unstructured LLM outputs, reformatting tool results, and dynamically constructing downstream prompts exemplifies the prompt-state management layer that distinguishes meta-harnesses from simple agent chains.