Skip to content
NVIDIA Nemotron Speech (Riva) NIM Operator

NVIDIA Nemotron Speech (Riva) NIM OperatorSkill

by NVIDIA
Released
4 views
v1.0.0
Apache-2.0
Repository Docs
markdown Development

Summary

NVIDIA's official skill for deploying and operating Nemotron Speech (Riva) NIMs — ASR, text-to-speech and translation, cloud-hosted or self-hosted on your own GPUs.

Features

  • Routes each request to a single reference playbook, keeping agent context tight
  • Covers ASR (Parakeet, Canary, Whisper, Nemotron ASR Streaming), TTS (Magpie) and NMT
  • Cloud inference via build.nvidia.com or self-hosted NIM containers on your own GPUs
  • Six-step GPU/VRAM/container readiness check before deployment
  • Custom .nemo to RMIR to NIM pipelines for fine-tuned ASR models and TTS voices
  • TTS pronunciation workflow: generate IPA candidates, test them, apply a custom dictionary
  • Treats NVIDIA_API_KEY and NGC_API_KEY as secrets and refuses to print or commit them

Install This Skill

Add this skill to your favorite AI agent in a few steps.

Any AI agent

This skill is plain instructions — it works with any assistant that accepts custom instructions or system prompts.

  1. Copy the skill content with the button below.
  2. Paste it into your agent's instruction file or system prompt (for example AGENTS.md, .cursorrules, or a custom instructions field).
  3. Ask the agent to apply the skill whenever the task matches.

Skill Content

Markdown Content

Copy this content and use it with your preferred AI agent

---
name: "nemotron-speech"
description: Routes NVIDIA Nemotron Speech (Riva) NIM tasks — deploys, runs, and tests ASR, TTS, and NMT NIMs on build.nvidia.com or self-hosted.
triggers:
  - Nemotron Speech
  - deploy Riva NIM
  - deploy ASR/TTS/NMT NIM
  - Riva ASR
  - Riva TTS
  - Riva translation
  - Parakeet
  - Canary
  - Whisper
  - Nemotron ASR Streaming
  - Magpie TTS
  - DNT tag
  - nemo2riva
  - riva-build
  - riva-deploy
  - RMIR
  - Riva NIM setup
  - NGC API key
  - force_eou
  - Silero VAD
  - Sortformer diarization
  - chunk size Riva
  - Riva HTTP
  - Riva WebSocket
  - grpc.nvcf.nvidia.com
  - build.nvidia.com Riva
  - custom TTS model
  - fine-tuned TTS
  - zero-shot TTS
  - zero-shot voice cloning
  - TTS pronunciation dictionary
  - SSML Riva
  - exaggeration_factor
  - OGG_OPUS TTS
  - TTS pipeline configuration
  - IPA pronunciation TTS
  - phoneme Riva TTS
  - custom TTS pronunciation
  - mispronounced word TTS
  - how to pronounce Riva TTS
  - TTS IPA candidates
version: "1.0.0"
license: Apache-2.0
metadata:
  author: "Nemotron Speech Team"
  team: riva
  tags:
    - nvidia
    - nemotron-speech
    - riva
    - nim
    - asr
    - tts
    - nmt
    - speech
    - speech-to-text
    - text-to-speech
    - translation
    - parakeet
    - canary
    - whisper
    - magpie
    - nemotron
    - grpc
    - http
    - websocket
    - cloud
    - nvcf
    - pronunciation
    - ipa
    - phoneme
  domain: ml
---

# Nemotron Speech Skills

> **Note:** "Nemotron Speech" is the public-facing name for what NVIDIA documents today as **Riva** / **Riva NIM**. All commands, container images, gRPC APIs, Python imports, and documentation URLs still use **"Riva"** — the rename is brand-only. Do not rename commands, images, or doc URLs.
>
> **Agent:** When walking the user through a multi-step workflow, announce each step before presenting it: **Step N/M — Step Title** (e.g., "**Step 1/4 — Deploy the Container**").

## Purpose

Single entry point for all NVIDIA Nemotron Speech (Riva) NIM workflows: ASR (speech-to-text), TTS (text-to-speech), and NMT (translation). Covers cloud-hosted inference via build.nvidia.com, self-hosted Docker deployment, client-protocol choice for ASR (gRPC, HTTP, WebSocket), custom NeMo model deployment via `riva-build`, ASR pipeline tuning (VAD, diarization, language models), and the prerequisite Docker / NGC / driver setup.

## When to Use This Skill

Use this skill for any Nemotron Speech / Riva NIM task — deployment, testing, custom model build, system requirements check, or model selection across ASR / TTS / NMT modalities.

## Workflow

Identify the user's task type, then load the corresponding reference file from `references/`. The reference files contain the detailed per-workflow content; this SKILL.md is a routing surface. Load only the reference relevant to the task at hand.

## Prerequisites

- For **self-hosted deployment**: NVIDIA AI Enterprise (NVAIE) entitlement, then complete the environment setup — NVIDIA drivers, Docker, Container Toolkit, NGC API key, Riva Python client. See [`references/setup.md`](references/setup.md).
- For **cloud-hosted inference**: `pip install -U nvidia-riva-client` and a valid `NVIDIA_API_KEY` from https://build.nvidia.com.
- Treat `NVIDIA_API_KEY` and `NGC_API_KEY` as secrets: never print, paste, commit, or log real key values. Prefer `--password-stdin` for Docker login and store persistent keys in a credential manager or a `chmod 600` env file rather than world-readable shell startup files.
- For **self-hosted Docker model caching**: host directories mounted at `/opt/nim/.cache` must be writable by the container user (the NIM container runs as `nvs:1000` internally), not just the host user. Run `sudo chown 1000:1000 $LOCAL_NIM_CACHE` after creating the directory so the container can write to it. Avoid world-writable modes — they let any local user replace cached model artifacts. Also avoid `-u "$(id -u):$(id -g)"` on the docker run — `/opt/nim/workspace` inside the container isn't writable to arbitrary UIDs. If you see `I/O error Permission denied (os error 13)` during model download, the host directory ownership is the issue.

## Instructions

- Match the user's task to one reference file and load only that file; the references are detailed, so progressive disclosure keeps context tight.
- Route setup requests for drivers, Docker, Container Toolkit, and NGC to [`references/setup.md`](references/setup.md).
- Route GPU compatibility, deployment readiness, and container health checks to [`references/deployment-readiness-checks.md`](references/deployment-readiness-checks.md).
- Route model choice across ASR, TTS, and NMT to [`references/model-selection.md`](references/model-selection.md).
- Route ASR deployment or inference for Parakeet, Canary, Whisper, and Nemotron ASR Streaming to [`references/asr.md`](references/asr.md).
- Route custom-trained NeMo ASR deployment (`.nemo` → RMIR → NIM) to [`references/asr-custom.md`](references/asr-custom.md).
- Route ASR pipeline configuration for VAD, diarization, language models, and chunk size to [`references/pipelines.md`](references/pipelines.md).
- Route TTS deployment or inference for Magpie to [`references/tts.md`](references/tts.md).
- Route custom or fine-tuned TTS model deployment (`.nemo` → RMIR → NIM) to [`references/tts-custom.md`](references/tts-custom.md).
- Route TTS synthesis pipeline configuration — SSML, zero-shot voice cloning, applying an existing pronunciation dictionary, audio encoding, sample rate, `custom_configuration` keys — to [`references/tts-pipelines.md`](references/tts-pipelines.md). *(For discovering and constructing the pronunciation itself, use `tts-pronunciation.md`.)*
- Route TTS pronunciation discovery — finding, testing, and applying IPA pronunciations for specific words or phrases — to [`references/tts-pronunciation.md`](references/tts-pronunciation.md).
- Route NMT deployment or inference for Riva Translate, language pairs, and DNT tags to [`references/nmt.md`](references/nmt.md).

## Source of truth

For per-release detail — current model catalog, container IDs, function IDs, voice lists, VRAM minimums, per-model feature support — **fetch or open the canonical NVIDIA doc** rather than relying on text in this SKILL.md or the references. Each reference file includes its own routing table to the relevant doc pages.

Top-level landing pages:

| Topic | URL |
|---|---|
| ASR support matrix | https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html |
| TTS support matrix | https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/tts.html |
| NMT support matrix | https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/nmt.html |
| Prerequisites (driver / GPU / OS) | https://docs.nvidia.com/nim/speech/latest/get-started/prerequisites.html |
| ASR pipeline configuration | https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html |
| ASR runtime customization | https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html |
| TTS custom deployment (`.nemo` / `.riva`, `riva-build`, RMIR) | https://docs.nvidia.com/nim/speech/latest/tts/custom-deployment.html |
| TTS request-time customization (SSML, pronunciation dictionaries, `custom_configuration`) | https://docs.nvidia.com/nim/speech/latest/tts/customization.html |
| TTS voices and emotional styles | https://docs.nvidia.com/nim/speech/latest/tts/voices.html |
| TTS zero-shot voice cloning | https://docs.nvidia.com/nim/speech/latest/tts/voice-cloning.html |
| TTS IPA phone set | https://docs.nvidia.com/nim/speech/latest/tts/phoneme-support.html |
| Cloud function IDs (per model) | `https://build.nvidia.com/<org>/<model>/api` |
| NGC model catalog | https://catalog.ngc.nvidia.com/models |

## Examples

**"Deploy a Parakeet ASR NIM"** → load [`references/asr.md`](references/asr.md), follow Option B (self-hosted), Steps 1–4.

**"Synthesize speech with Magpie"** → load [`references/tts.md`](references/tts.md), follow Option A (cloud) or Option B (self-hosted).

**"Translate English to German"** → load [`references/nmt.md`](references/nmt.md), follow the 4-step flow.

**"Convert my fine-tuned `.nemo` to a NIM"** → load [`references/asr-custom.md`](references/asr-custom.md) for the 4-phase pipeline and [`references/pipelines.md`](references/pipelines.md) for build-time config.

**"Deploy a custom fine-tuned TTS voice as a NIM"** → load [`references/tts-custom.md`](references/tts-custom.md) for the 4-phase pipeline.

**"Use zero-shot voice cloning with Magpie"** → load [`references/tts-pipelines.md`](references/tts-pipelines.md).

**"Add SSML emphasis tags to my TTS request"** → load [`references/tts-pipelines.md`](references/tts-pipelines.md).

**"'NVIDIA' sounds wrong in my Magpie TTS output — suggest a few IPA options to test"** → load [`references/tts-pronunciation.md`](references/tts-pronunciation.md), generate IPA candidates, synthesize variants, then output all three delivery formats.

**"How do I fix the pronunciation of 'NIM' in Riva TTS with a custom_dictionary in gRPC Python?"** → load [`references/tts-pronunciation.md`](references/tts-pronunciation.md), propose IPA for 'NIM', show wire format and gRPC snippet.

**"Can my GPU run this?"** → load [`references/deployment-readiness-checks.md`](references/deployment-readiness-checks.md) and run the 6-step system check.

**"Which Riva model should I use?"** → load [`references/model-selection.md`](references/model-selection.md), apply the decision framework, then fetch the support matrix for the specific current model name.

## Naming & Terminology

- **Skill brand**: Nemotron Speech (public-facing name).
- **Internal naming preserved**: commands (`riva-build`, `riva-deploy`, `riva_streaming_asr_client`), Python client (`riva.client`), gRPC namespace (`nvidia.riva.asr.*`), container registry (`nvcr.io/nim/nvidia/*`), and all NVIDIA documentation URLs still use **"Riva"**. Do not rename these in code, commands, or docs.

## Troubleshooting

For task-specific runtime or modality issues, use the relevant reference file (`references/<task>.md`). Cross-cutting readiness checks:

- **Container does not become ready** → [`references/deployment-readiness-checks.md`](references/deployment-readiness-checks.md) (system check + health check table)
- **Health check fails** → [`references/deployment-readiness-checks.md`](references/deployment-readiness-checks.md)
- **`docker pull` from `nvcr.io` returns 403** → [`references/setup.md`](references/setup.md) (Step 5 — Docker login)
- **Wrong base image / model architecture mismatch** → [`references/asr-custom.md`](references/asr-custom.md) (Phase 2 base image)
- **VRAM / GPU compatibility** → [`references/deployment-readiness-checks.md`](references/deployment-readiness-checks.md), then verify on the support matrix

## Limitations

- x86_64 architecture only — WSL2 on Windows requires Podman and supports a subset of NIMs (see [`references/setup.md`](references/setup.md))
- Self-hosted deployment requires an NVIDIA AI Enterprise license
- Cloud-hosted inference requires an active `NVIDIA_API_KEY` and internet access
- Public skill branding is **"Nemotron Speech"**; commands, container images, Python imports (`riva.client`), gRPC services (`nvidia.riva.*`), and NVIDIA documentation URLs still use **"Riva"** — follow official docs and catalogs for naming, do not rename these in commands or code

## Next Steps

- Verify hardware compatibility: [`references/deployment-readiness-checks.md`](references/deployment-readiness-checks.md)
- Set up the environment: [`references/setup.md`](references/setup.md)
- Pick a model: [`references/model-selection.md`](references/model-selection.md)
- Deploy: [`references/asr.md`](references/asr.md), [`references/tts.md`](references/tts.md), or [`references/nmt.md`](references/nmt.md)

Usage Instructions

Learn how to use this skill with different AI agents.

Generic Instructions

Install with npx skills add nvidia/skills --skill nemotron-speech, or copy the skills/nemotron-speech directory from github.com/NVIDIA/skills into your agent's skills folder. Supported agents include Claude Code, Codex, Cursor and Kiro.

Example Usage

Deploy a Parakeet ASR NIM on my local GPU and transcribe sample.wav

Description

NVIDIA's speech stack is one of the more capable and one of the more fiddly things to stand up: Parakeet, Canary, Whisper and Nemotron ASR Streaming for recognition, Magpie for synthesis, Riva Translate for NMT — each with its own container, function ID, VRAM floor and support matrix. This official NVIDIA skill gives a coding agent the routing knowledge to work through that without you memorising it.

How it is built

The SKILL.md itself is a routing surface, not a manual. It matches the request to one reference file and loads only that file — a deliberate progressive-disclosure design that keeps context tight when the underlying material runs to a dozen detailed playbooks. Ask to deploy a Parakeet ASR NIM and it loads the ASR reference and walks Steps 1-4; ask which model to use and it loads the model-selection decision framework instead.

What it covers
  • Environment setup — drivers, Docker, the NVIDIA Container Toolkit, NGC login, and the Riva Python client.
  • Readiness checks — a six-step system check for GPU compatibility, VRAM and container health before you waste an hour on a pull that was never going to run.
  • Model selection across ASR, TTS and NMT.
  • ASR — deployment and inference for Parakeet, Canary, Whisper and Nemotron ASR Streaming, plus pipeline configuration for VAD, Sortformer diarization, language models and chunk size.
  • TTS — Magpie deployment, SSML, zero-shot voice cloning, audio encoding, and a dedicated workflow for discovering and testing IPA pronunciations when a word comes out wrong.
  • NMT — Riva Translate, language pairs and DNT tags.
  • Custom models — the four-phase .nemo → RMIR → NIM pipeline for both fine-tuned ASR and fine-tuned TTS voices.
Naming, because it will confuse you

"Nemotron Speech" is the public brand for what NVIDIA's documentation still calls Riva. Commands (riva-build, riva-deploy), the Python client (riva.client), the gRPC namespace and every doc URL keep the Riva name — the rename is brand-only, and the skill is explicit that an agent must not rewrite them.

Constraints

x86_64 only; WSL2 needs Podman and supports a subset of NIMs. Self-hosted deployment requires an NVIDIA AI Enterprise entitlement, while cloud-hosted inference through build.nvidia.com needs only an NVIDIA_API_KEY. The skill treats NVIDIA_API_KEY and NGC_API_KEY as secrets and instructs the agent never to print, paste, commit or log them — worth knowing, because an agent that echoes an NGC key into a terminal transcript has effectively published it. Apache-2.0.

Related Skills

Skill: NVIDIA Warp Gradient Debugging

by NVIDIA

New

Diagnoses wrong gradients in differentiable NVIDIA Warp programs by measuring first — comparing autodiff against finite differences on a shrunk reproduction before proposing any fix.

Development
3 views

Skill: Google Workspace CLI (gws)

by Google Workspace

New

Google's official skill for the gws CLI — drive Gmail, Drive, Calendar, Sheets, Docs, Chat and Admin APIs from an agent, with Model Armor screening.

Development
4 views 1 copies

Skill: Netlify Database

by Netlify

New

Netlify's official skill for zero-config managed Postgres — querying from Functions, Drizzle setup, migrations and per-preview database branches.

Development
3 views

Skill: WordPress Block Development

by WordPress

New

Official WordPress skill for Gutenberg block work: block.json, attributes and serialization, dynamic rendering, and the deprecation path that keeps existing content valid.

Development
3 views 1 copies
Browse all skills →