Skip to content

CReFT-CAD: Boosting Orthographic Projection Reasoning for CAD via Reinforcement Fine-Tuning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CReFT-CAD applies curriculum-driven reinforcement fine-tuning (RL) followed by supervised post-tuning to teach VLMs precise orthographic projection reasoning for CAD drawings. The authors also release TriView2CAD, the first large-scale benchmark (200K synthetic + 3K real-world samples) for this task. On real-world OOD data, CReFT-CAD reaches 46.67% accuracy vs. 36.15% for standard SFT and 13.47% for the untuned baseline.

Problem

Standard 3D reconstruction pipelines for CAD introduce pixel-level discretization errors and lack parametric editability. VLMs fine-tuned with vanilla SFT on orthographic projection reasoning devolve into pattern memorization, performing poorly on out-of-distribution (OOD) real-world drawings that feature occlusions, redundant annotation lines, and non-numeric labels. No open-source benchmark exists with precise dimensional annotations, logical reasoning tasks, and multi-modal CAD representations.

Method

CReFT-CAD is a two-stage paradigm built on GOT-OCR2.0 (ViT encoder + Qwen2.5 LLM):

Stage 1 — Curriculum-driven Reinforcement Fine-tuning (CReFT): Three tasks of increasing difficulty, each trained sequentially via GRPO: 1. Dichotomous choice: binary yes/no whether all 15 parameters match; reward = 1 iff fully correct. 2. Multiple choice: select correct parameter lists from masked distractors; partial reward (0.2) for selecting a subset of correct lists. 3. Parameterization with CoT: predict all 15 parameters via chain-of-thought; difficulty-aware rewards (1/1.5/2 for easy/medium/hard attributes). Each task begins with a small warm-up SFT phase to initialize the RL reward signal.

Stage 2 — Supervised Post-tuning: Multi-task VQA SFT (full parameter list output, parameter validation, view matching, component counting) to refine instruction-following on real-world interactive CAD queries.

TriView2CAD is constructed via a constraint-driven pipeline: (1) sample 15-dimensional parameter vectors subject to intra-/inter-view geometric constraints; (2) render DXF vector drawings and PNG rasters using ezdxf + FreeCAD; (3) generate STEP/B-Rep 3D models via FreeCAD Python API.

Key Contributions

  • CReFT-CAD: two-stage RL + SFT fine-tuning paradigm with curriculum difficulty ordering and difficulty-aware rewards for orthographic projection reasoning.
  • TriView2CAD benchmark: 200K synthetic + 3K real-world three-view CAD samples with 15-dimensional precise dimensional annotations and six interoperable modalities (JSON, DXF, PNG, scripts, STEP, B-Rep).
  • Comprehensive benchmarking of 10 leading VLMs (GPT-4o, Gemini 2.5 Pro, Claude 4 Vision, Qwen2.5-VL, InternVL3, etc.) on orthographic projection reasoning across four prompt formats with/without reasoning guidance.
  • Ablation demonstrating CoT is essential: Task 1+2+3 with CoT reaches 81.35% vs. 46.24% without CoT on the in-domain set.

Results

  • In-domain (synthetic test set): CReFT-CAD achieves 84.03% accuracy ("+Attribute Explanation" prompt with reasoning guidance), vs. best VLM baseline GPT-4o at 39.71% and Qwen2.5-VL at 38.88%.
  • Under "Test Image Only" (no extra context): CReFT-CAD reaches 80.86%, vs. GPT-4o 26.06%, Qwen2.5-VL 24.54%.
  • OOD real-world test set (3K samples):
  • Untuned Qwen2.5-VL: 13.47%
  • Standard SFT: 36.15%
  • CReFT-CAD: 46.67%
  • Ablation (in-domain, CoT vs. no-CoT):
  • Task 3 alone: 46.16% → 74.15% with CoT
  • All three tasks: 46.24% → 81.35% with CoT

Limitations

  • Real-world OOD accuracy (46.67%) remains far below in-domain performance (84%), leaving substantial headroom for generalization improvement.
  • Dataset domain is restricted to prefabricated bridge pier CAD drawings; generalization to other engineering domains (mechanical parts, architectural plans) is untested.
  • The RL warm-up SFT requirement per task adds training complexity; reward remains zero without it, indicating brittleness of the RL initialization.
  • Task 2 (multiple-choice with masking) alone harms performance when added incrementally before Task 3; it only helps when all three tasks are combined.

Relevance to Vision-Language Models

CReFT-CAD is a concrete demonstration that GRPO-style reinforcement fine-tuning, already proven for text-only reasoning (DeepSeek-R1), transfers effectively to vision-language tasks requiring precise quantitative extraction from specialized visual inputs (engineering drawings). The curriculum design—progressively harder tasks with difficulty-aware rewards—offers a transferable recipe for any VLM application where SFT memorization is a known failure mode. The paper also surfaces a significant benchmark gap: even frontier VLMs (GPT-4o, Gemini 2.5 Pro) score below 40% on a domain requiring cross-view geometric consistency and composite arithmetic, pointing to fundamental limits in current VLM spatial-quantitative reasoning. For VLM researchers, TriView2CAD provides a rare benchmark with ground-truth dimensional precision rather than categorical labels.