Skip to content

Unified Audio Intelligence Without Regressing on Text Intelligence

🕒 Published (v1): 2026-07-06 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Audex (Nemotron-Labs-Audex-30B-A3B) is a unified audio-text MoE LLM (30B total, 3B activated parameters) built on the Nemotron-Cascade-2 text-only backbone that achieves state-of-the-art audio understanding and generation while preserving near-identical performance on reasoning, knowledge, alignment, and agentic text benchmarks. It addresses the well-documented phenomenon where adding multimodal generation to LLMs degrades text intelligence. A single Transformer decoder handles both modalities by projecting audio into the text embedding space and treating quantized audio tokens uniformly with text tokens.

Problem

Extending LLMs to support multimodal inputs and especially multimodal outputs (audio generation, speech synthesis) systematically degrades text-benchmark performance, including reasoning and agentic capabilities such as tool use. For example, Qwen3-Omni-30B-A3B-Thinking shows noticeable regression versus its text-only backbone Qwen3-30B-A3B-Thinking-2507 on reasoning benchmarks; Qwen3.5-Omni-Flash similarly regresses versus Qwen3.5-35B-A3B. Open frontier LLMs (DeepSeek-V4, GLM-5.1) remain text-only to avoid this tradeoff. The challenge is building a native multimodal audio LLM that retains the full text intelligence of its backbone.

Method

Architecture: A single MoE Transformer decoder (Nemotron-Cascade-2-30B-A3B backbone: 52 layers, dim 2688, 128 routable/6 activated experts in a hybrid Mamba2-Transformer). Audio inputs pass through AF-Whisper (Whisper Large-v3 finetuned for general audio, 25 Hz output at dim 1280) and 2-layer MLP adapters that project into the LLM embedding space. The vocabulary is extended from 131,072 to 205,312 to include discrete audio tokens: X-Codec2 (FSQ, 65,536 vocab, 50 tok/s) for speech and X-Codec (RVQ first 4 of 8 layers, 4,096 vocab, \(50 \times 4 = 200\) tok/s) for non-speech audio. Audio tokens are generated autoregressively with the same cross-entropy loss as text tokens (Eq. 1), using square-averaging loss (Eq. 2) over packed sequences.

Classifier-Free Guidance (CFG): For audio generation, unconditional distribution \(p_\theta(\cdot|\emptyset)\) is learned by pre-processing 10% of speech/audio generation samples to replace captions with padding tokens. CFG sampling modifies logits as \(p^{\text{CFG}}_\theta(\cdot|x_{<i}) = p_\theta(\cdot|x_{<i}) + (\lambda - 1)[p_\theta(\cdot|x_{<i}) - p_\theta(\cdot|\emptyset)]\).

Training pipeline (Figure 3): 1. Reuse Nemotron-Cascade-2 SFT checkpoint (text-only). 2. Audio warmup: freeze text token embeddings, train only MLP adapters and audio token embeddings. 3. Audio generation SFT: unfreeze full LLM, train on speech/audio generation + text-only data. 4. Full unified SFT: unfreeze MLP adapters, add audio understanding / ASR / AST tasks. 5. Cascade RL (text-only, identical to Nemotron-Cascade-2) + multi-domain on-policy distillation (MOPD).

An alternative single-stage consolidated SFT combines all tasks simultaneously. Total training data: 394M samples, 1,092K audio hours, 157.4B audio tokens + 320.5B text tokens.

Key Contributions

  • Audex-30B-A3B: first open frontier unified audio-text MoE LLM with near-zero text-benchmark regression despite supporting audio/speech input and output.
  • Identification and resolution of text embedding degradation: freezing text token embeddings during audio warmup prevents quality loss.
  • Text-only Cascade RL applied post-SFT to recover and maintain text intelligence (reasoning, agentic) after multimodal training.
  • Square-averaging loss (Eq. 2) for stable training over packed multi-modal sequences.
  • Two SFT curricula studied (multi-stage vs. single-stage consolidated), showing both are viable without audio pretraining.
  • Separate speech (X-Codec2/FSQ) and non-speech (X-Codec/RVQ4) codecs with dedicated vocabularies and decoders (streaming ConvNeXt for speech; enhancement VAE for non-speech).
  • Release of model checkpoints for open research; only open model supporting general audio generation beyond speech.

Results

All numbers from Table 1 comparing Audex-30B-A3B to its backbone Nemotron-Cascade-2-30B-A3B and open/proprietary baselines:

Text/agentic (vs. backbone, marginal regression): - AIME 2025: 91.2 vs. backbone 92.4; AIME 2026: 89.4 vs. 90.9; HMMT Feb25: 92.2 vs. 94.6 - IMO AnswerBench: 81.1 vs. 79.3 (Audex slightly higher) - LiveCodeBench v6: 85.3 vs. 87.2; SciCode: 35.2 vs. 36.4 - \(\tau^2\)-Bench (agentic): 57.2 vs. 58.9; Terminal Bench 2.0: 19.1 vs. 21.1; SWE Verified: 48.2 vs. 50.2 - Qwen3-Omni shows substantially larger regression vs. Qwen3-30B-A3B-Thinking (e.g., AIME 2025: 73.7 vs. 85.0)

Audio/speech: - MMAU (audio understanding): 75.6 — comparable to Step-Audio-R1.1-33B (73.6) and Qwen3-Omni (75.4); below proprietary Qwen3.5-Omni-Flash (80.4) - LibriSpeech clean WER: 1.34 (vs. Step-Audio 1.66, Qwen3-Omni 2.22, Qwen3.5-Omni-Flash 1.11) - OpenASR WER: 6.82 (best among open models) - Speech-to-speech (BigBenchAudio): 90.0 vs. Qwen3.5-Omni-Flash proprietary 59.0 - Audio generation (AudioCaps OpenL3 Fréchet Distance): 66.9; SongDescriber: 62.7 — no open-model baseline available

Limitations

  • Marginal but non-zero regression on most text benchmarks (e.g., SWE Verified −2 pp, \(\tau^2\)-Bench −1.7 pp, ArenaHard v2 −1.9 pp, GPQA-Diamond −1.2 pp).
  • Audio understanding still lags proprietary Qwen3.5-Omni-Flash (MMAU 75.6 vs. 80.4, MMAR 63.2 vs. 74.0).
  • Text-to-audio generation output is fixed at 10 seconds during training due to instability with variable lengths.
  • Non-speech audio at 200 tok/s (RVQ4 flattening) is computationally expensive for long audio.
  • No audio-specific RL (only text-domain RL is applied post-SFT); audio generation quality may be capped by SFT data quality.
  • Model size (30B total / 3B active) limits accessibility compared to smaller open alternatives.

Relevance to Agentic AI / LLM Agents

Audex directly addresses one of the core risks for multimodal agentic systems: the erosion of tool-use, coding, and planning capabilities when a backbone LLM is extended with new modalities. The near-zero regression on \(\tau^2\)-Bench, Terminal Bench 2.0, and SWE Verified demonstrates that text-only post-training RL (applied after audio SFT) is an effective recipe for preserving agentic intelligence in a unified model. This is practically relevant for building voice-driven or audio-grounded agents that must also perform multi-step reasoning, call tools, and complete software tasks without degrading on those capabilities. The architecture — a single decoder with uniform token treatment — also simplifies deployment of agentic pipelines that interleave audio perception and text-based reasoning without cascading multiple specialized models.