NVIDIA 發佈 Nemotron 3.5 Transcribe 開放權重語音模型及 Granary v2 數據集

NVIDIA released Nemotron 3.5 Transcribe on October 28, 2026, an open-weight speech model that combines multilingual transcription, speaker attribution, translation, and audio question answering in a single checkpoint, alongside Granary v2, an open speech dataset and curation pipeline whose public manifest covers 463,062 hours across 48 languages.

The model transcribes 40 languages, translates 31 of them to or from English, and attributes speakers in 13, according to NVIDIA’s announcement on Hugging Face. It handles recordings up to one hour long and labels up to eight speakers. The model card lists an October 28, 2026 release date across Hugging Face, GitHub via NVIDIA NeMo Speech, NGC, and Build.Nvidia.com, with weights released under the OpenMDW-1.1 license and the model ready for commercial or non-commercial use. Hosted endpoints are offered by Baseten, DeepInfra, and Simplismart, with an interactive demo on Hugging Face Spaces and ModelScope Studios.

Architecture and Prompting

Transcribe pairs a speech encoder with a language backbone in one 34B-parameter checkpoint, of which about 4.3B parameters are active per token, with a 65,536-token context. The speech side combines a 600M-parameter Transformer ASR encoder, the roughly 99M-parameter Nemotron-3-Diarization speaker encoder, and a connectionist temporal classification head for word timestamps; the language side is NVIDIA’s Nemotron-3.5 Lightning 30B-A3B, a hybrid Mamba-2–Transformer mixture-of-experts model. Weights ship in BF16, with FP8 and NVFP4 quantized variants and an optional DFlash2 draft model for speculative decoding.

Natural-language prompts control the output: developers can request verbatim or clean written transcripts, supply context vocabulary such as product or medical terms, ask for speaker tags and word-level timestamps, translate speech to or from English, or pose questions about the audio. Audio question answering is trained in English, and the card’s language matrix grades each of the 40 languages by task with quality levels from Leading to Supported, noting that audio understanding and spoken instruction following were evaluated only in English. The model card stresses that Transcribe is an offline model: internal audio chunking does not make it a real-time streaming service.

Training used approximately 1.72 million unique indexed audio hours across about 372 million examples, plus 5.6 billion unique text tokens, drawing on Granary v2, multi-speaker meeting corpora, audio-understanding data, and synthetic PII-aware speech. Some training data was licensed from David AI; the card states that David AI’s DAI-ASR-I18N evaluation set was not part of training.

Benchmark Results NVIDIA Reports

NVIDIA reports that Transcribe ranks first among open-weight models on the Hugging Face Open ASR English leaderboard at 5.33% average word error rate, with API models excluded, and posts the lowest multilingual result, 4.62% mean WER across seven languages, and the lowest long-form result, 10.98%, among the open-weight models in the model card’s comparison. In the card’s own tables, captured October 12, 2026, the nine-dataset English mean is 4.71% WER against 4.94% for Qwen3-ASR-1.7B, the next-best open-weight entry, and the multi-speaker mean cpWER is 15.80 against 16.83 for DiCoW v3.3 paired with DiariZen.

On Artificial Analysis’s speech-to-text index, NVIDIA says Transcribe ties for first among open-weight models at 2.8% AA-WER while activating about 4B parameters per token, versus 24B for the model it ties. For speaker attribution, the announcement reports the lowest diarization error rate among the 14 systems on David AI’s 21-language DAI-ASR-I18N diarization track at 18.26%, slightly ahead of NVIDIA’s own Nemotron 3 Diarization at 18.38%, and first place among 14 systems on VoiceArena’s English Diarization-Bench at 32.4% cpWER, with deletions of 8.32% of reference words against a median of about 31%.

Granary V2 Dataset and Pipeline

Granary v2, released alongside the model, is the open data effort behind its training. Its public manifest spans 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1; NVIDIA notes these figures describe the dataset release rather than Transcribe’s full training mixture. Built with NVIDIA NeMo Curator, the pipeline generates three transcript forms per recording (verbatim with text normalization, inverse-normalized with disfluencies, and inverse-normalized without disfluencies) plus captions, code-switching restoration, speech-grounded question-answer pairs, contextual biasing terms, and translations with automatic quality scores.

The pipeline runs in four stages: metadata and audio preparation; transcription with primary and recovery ASR models gated by a hallucination check; task-label generation with agreement checks; and a high-fidelity filter that, for the VoxPopuli corpus, reduced 354,000 hours to 111,000 hours across 22 European languages. Manifests carry entity hints, instruction pairs, provenance, and per-record quality flags; the source audio is downloaded separately under each corpus’s original license, and the pipeline runs with Ray and NVIDIA Dynamo backends.

Medical Fine-Tuning Case Study and Deployment

To demonstrate adaptation, NVIDIA’s announcement describes a medical case study on Eka Care’s open benchmark of 3,619 recordings from Indian doctors and medical students. Naming English and the Latin alphabet in the prompt cut full-benchmark WER from 30.8% to 23.8%, and fine-tuning brought it to 11.8%. On 2,764 held-out clips scored with Eka Care’s KARMA pipeline, fine-tuning reduced WER from 22.44% to 12.17%, a 45.8% relative reduction; keyword WER fell from 9.2% to 6.5% and semantic WER from 10.0% to 7.6%, while LibriSpeech test-clean WER rose slightly from 1.54% to 1.69%. The adaptation combined about 153 hours of synthetic speech with 29 hours of accented English from Common Voice and 63 hours of medical recordings from MultiMed, and took roughly 14 GPU-hours on one NVIDIA GB300.

The model runs on a single NVIDIA GPU, from Jetson AGX Thor and DGX Spark to Blackwell data-center systems. NVIDIA reports that with the optimized NIM implementation, one RTX PRO 6000 Blackwell Server Edition GPU transcribes up to 2,238 seconds of audio per second using NVFP4 precision with DFlash2 speculative decoding, and the announcement says the model processes 100 hours of audio in under three minutes in NVFP4 on 30-second clips at high concurrency; the model card notes these serving optimizations are not yet part of the open-source release. Serving uses vLLM 0.32.0 with the NeMo Speech plugin or local inference with NeMo-Speech.cpp, and a fine-tuning agent skill lets a coding agent run the adaptation workflow, with ModelScope supporting fine-tuning through its ms-swift framework.