Skip to content

NOMAD: A Multi-Agent LLM System for UML Class Diagram Generation from Natural Language Requirements

🕒 Published (v1): 2025-11-27 12:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NOMAD is a sequential five-agent pipeline—Concept Extractor → Relationship Comprehender → Model Integrator → Code Articulator → optional Validator—that decomposes UML class diagram generation from natural language requirements into role-specialized subtasks with constrained output schemas. It outperforms single-agent LLM baselines on precision/recall/F1 for classes, attributes, and relationships, while surfacing persistent challenges in attribute extraction. The paper also introduces the first error taxonomy for LLM-generated UML diagrams.

Problem

Single LLMs generating UML class diagrams from natural language suffer from semantic degradation at scale (serious errors reported for diagrams with >10–12 classes), poor relationship inference, and incomplete attribute extraction. Rule-based NLP pipelines (POS tagging, dependency parsing, heuristic ontology mapping) are brittle and domain-specific, failing on linguistically ambiguous or diverse requirement formats. No prior work applied modular multi-agent decomposition to this structured-artefact generation task.

Method

NOMAD defines a fixed sequential pipeline of four generation agents plus one optional verification agent, all implemented in LangChain with GPT-4o at temperature 0:

  1. Concept Extractor: identifies UML classes and their attributes from requirements text.
  2. Relationship Comprehender: receives extracted entities and infers associations, aggregations, compositions, and generalizations.
  3. Model Integrator: merges entities and relationships into a validated JSON intermediate representation, enforcing a strict schema to eliminate residual natural-language ambiguity.
  4. Code Articulator: translates the JSON into syntactically correct PlantUML, using an LLM (rather than hard-coded rules) to handle JSON variations and maintain stylistic coherence.
  5. Validator (optional): a reflective agent that inspects the generated diagram against original requirements without presupposing errors—it autonomously decides whether revisions are warranted.

Evaluation uses automated PlantUML parsing (regex) with NLTK normalization for naming variants, computing precision/recall/F1 over classes, attributes, and relationships. Relationship matching is performed under two criteria: hard (exact source, target, and type) and soft (source and target connected, regardless of type/direction). Two datasets are used: a Northwind 2.0 benchmark (21 classes, 212 total elements, reverse-engineered from a relational schema with rule-based NL generation) and eight human-authored UML exercises (UC1–UC8, 8–25 elements each) from prior work.

Key Contributions

  • NOMAD framework: modular sequential multi-agent pipeline for NL→UML class diagram generation with constrained per-agent output schemas.
  • Northwind 2.0 benchmark: a gold-standard case study with 21 classes and 212 UML elements, including reverse-engineered diagram and curated NL requirements, released publicly.
  • Automatic evaluation framework: PlantUML-parsing + NLTK-normalized F1 with hard/soft relationship matching, enabling reproducible benchmarking.
  • First error taxonomy for LLM-generated UML: hierarchical classification across three dimensions—Classes (missing, extra, misrepresented), Attributes (missing, extra, wrong), Relationships (missing, extra, duplicate, misclassified)—covering only logical/semantic errors since all generated diagrams were syntactically valid.
  • Verification study: Validator agent evaluated across GPT-4o and DeepSeek-V3, framing verification as a design probe rather than a definitive fix.

Results

The results section is truncated in the provided text before specific F1 numbers are reported. From what is stated: - NOMAD outperforms all single-agent baselines on the breadth evaluation (8 use cases) and the Northwind case study. - Persistent degradation observed in fine-grained attribute extraction across all configurations. - Verification (Validator agent) shows mixed effects: improvements in some configurations, limited or no benefit in others; consistent behavior tested across GPT-4o and DeepSeek-V3. - All generated diagrams were syntactically correct (PlantUML validity), so taxonomy covers only semantic/logical errors.

(Specific F1 values are not available in the provided excerpt.)

Limitations

  • Attribute extraction remains a persistent failure mode unresolved by the multi-agent decomposition.
  • The Validator agent shows inconsistent gains; single-shot verification without inter-agent feedback or iterative correction is insufficient for reliable error recovery.
  • The pipeline is strictly sequential with no inter-agent feedback loops or planning; inter-agent feedback is acknowledged but deferred as future work.
  • Northwind NL requirements are generated by a rule-based script from a schema, not written by domain experts, potentially introducing systematic artifacts favoring structured enumeration.
  • Experiments use only GPT-4o (and DeepSeek-V3 for verification); generalization to smaller or open-source models for full pipeline runs is not evaluated.
  • No evaluation of latency, token cost, or scalability beyond 21-class diagrams.

Relevance to Harnesses / Meta-Harnesses

NOMAD is a concrete realization of the sequential pipeline harness pattern: each agent has a typed input/output contract (JSON schema, PlantUML syntax), making the pipeline composable and individually debuggable—directly relevant to harness design principles of modularity and observability. The JSON intermediate representation as a stage-boundary interface is an architectural pattern worth noting for multi-stage harness design. The Validator's treatment as an optional, swappable "design probe" rather than a hard pipeline stage exemplifies the kind of configurable verification slot that meta-harness frameworks need to support. The paper's hybrid evaluation harness—automated PlantUML parsing + NLTK normalization + dual matching criteria—is itself a reusable evaluation sub-harness applicable to other structured-artefact generation pipelines.