A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency¶

🕒 Published (v1): 2026-04-24 08:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This study systematically evaluates how model scale affects downstream clinical prediction performance for encoder-only Transformer foundation models pretrained on 2.3 million patients from a 519-hospital Japanese claims database. The key finding is that downstream performance saturates at task-dependent thresholds—medication prediction saturates at 11M parameters while disease prediction benefits from models up to 101M—challenging the "bigger is always better" assumption prevalent in medical AI. Task-optimal models consistently outperform LightGBM baselines despite using only count-aggregated features.

Problem¶

Prior structured medical foundation models (BEHRT, Med-BERT, TransformEHR) evaluate only a single model size, leaving the performance–cost tradeoff across scales unexamined. Whether NLP scaling laws (monotonically decreasing loss → monotonically improving downstream performance) transfer to structured EHR data—with its limited vocabulary (~1,930 codes) and sparse observations—is unknown, yet drives decisions to adopt increasingly large models with high pretraining costs.

Method¶

Data: Anonymized outpatient/inpatient claims (MDV; April 2014–November 2024) from 2,294,687 patients across 32 hospitals sampled from 519 in Japan. ICD-10 diagnoses (936 codes after frequency filtering) and YJ medication codes (994 codes, 7-digit prefix) form a vocabulary of ~1,930 tokens.

Sequence construction: Each patient is a chronologically ordered token sequence [CLS], [sex], (code, age_in_days), ... up to length 8,000. Continuous age is embedded via Piecewise Linear Encoding (PLE) with quantile-defined intervals, combined with separate categorical embedding matrices for special tokens, ICD-10, and YJ codes.

Pretraining: Masked language modeling (MLM) with 15% masking; loss is the unweighted sum of cross-entropy for categorical codes and MSE for age-in-days. Five encoder-only Transformers trained: 2.2M, 4.7M, 11M, 32M, 101M parameters (hidden dims 128–2048, fixed 4 layers, 16 heads).

Fine-tuning: CLS-token hidden state → sigmoid linear head for binary 1-year prediction. Four tasks: primary hypertension (I10, 13.8% positive), chronic kidney disease (N189, 2.63%), amlodipine initiation (14.9%), pregabalin initiation (6.93%). Fine-tuned with 100/500/1,000 labeled patients (1:1 balanced). Compared against from-scratch (randomly initialized) models of identical architecture and an LGBM baseline using 1,932 count-aggregated features.

Key Contributions¶

Systematic five-scale evaluation (2.2M–101M) of structured medical foundation models on a nationwide Japanese claims database, the first to characterize task-dependent capacity ceilings for encoder-only architectures with supervised fine-tuning.
Empirical demonstration that pretraining loss decreases monotonically with scale, but downstream AUPRC saturates at task-dependent thresholds: disease prediction benefits from 32M–101M; medication prediction saturates at 11M (76% pretraining time reduction: 53.9 h vs. 232.2 h).
Task-optimal pretrained Transformer consistently outperforms LGBM in AUPRC across all four tasks under limited-label conditions (100–1,000 patients), justifying pretraining cost when model size is matched to task.
Practical guidance linking task regularity (guideline-driven medication vs. complex disease progression) to appropriate model capacity for structured medical data.

Results¶

Pretraining loss: Decreases monotonically from 8.22 (2.2M) to 5.74 (101M); computational cost grows from 68.9 to 1,636.5 GFLOPs; wall-clock time from 14.4 h to 232.2 h.
Disease prediction (hypertension, CKD): Optimal models are 32M or 101M; AUPRC improvements continue beyond 11M.
Medication prediction (amlodipine, pregabalin): Performance saturates at 11M; scaling to 101M (4.3× more compute) yields no AUPRC gain.
Pretrained vs. from-scratch: Pretrained models generally outperform from-scratch counterparts; gains are more consistent and larger in AUPRC than AUROC (relevant given 2.63%–14.9% class imbalance).
vs. LGBM: Task-optimal pretrained Transformer exceeds LGBM AUPRC on all four tasks across all three label-count conditions (100/500/1,000 patients).

Limitations¶

Claims/DPC data only; no laboratory values, vital signs, or clinical notes, which could enrich representations.
Single-database evaluation (32 of 519 hospitals); no external validation or prospective testing.
No guaranteed cross-facility longitudinal linkage for individual patients within the database.
Tasks restricted to 1-year disease onset and medication initiation; saturation behavior may differ for readmission, mortality, or other endpoints.
Encoder-only discriminative architecture; findings may not generalize to generative/decoder-only models, which show continued gains with scale on autoregressive event simulation tasks.

Relevance to Foundation Models in Medicine¶

This paper directly addresses a fundamental design question for clinical foundation models: whether the NLP scaling paradigm applies to structured EHR data. The task-dependent saturation result—medication prediction plateauing at 11M parameters—provides empirical evidence that structured medical data's limited vocabulary and sparsity impose representation capacity ceilings absent in free-text models, complicating direct transfer of scaling intuitions from GatorTron, NYUTron, or Med-PaLM. For practitioners building or deploying medical foundation models (e.g., BEHRT-style architectures), the finding that a task-appropriate mid-scale model beats both from-scratch models and strong GBDT baselines under limited labels is practically significant, offering a principled compute-efficiency framework. The work also highlights that architectural choice (discriminative vs. generative) interacts with scaling behavior, a dimension underexplored in the structured EHR literature.