Skip to content
NVIDIA VSS Video Q&A

NVIDIA VSS Video Q&A Skill

NVIDIA's official skill for asking a vision-language model fresh questions about recorded video through the Video Search and Summarization blueprint — only when the pixels actually need looking at.

GitHub Docs

Install

Install with the skills CLI

One command for Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf and dozens of other agents. Needs Node.js; the skill is fetched straight from NVIDIA/skills.

$ npx skills add NVIDIA/skills --skill vss-ask-video

Run it in a terminal. The CLI finds the agents installed on your machine, asks which ones to add the skill to, and whether to install it into the folder you ran it from or for all your projects.

Other ways to install

Copy the whole skill folder — SKILL.md and any scripts or references beside it — into ~/.claude/skills/vss-ask-video/, where Claude Code picks it up in every project. Only git is needed.

tmp=$(mktemp -d) && \
  git clone --depth 1 --branch main https://github.com/NVIDIA/skills.git "$tmp" && \
  mkdir -p ~/.claude/skills && \
  cp -R "$tmp"/skills/vss-ask-video ~/.claude/skills/vss-ask-video

No scripts or references next to it? Saving the SKILL.md alone as ~/.claude/skills/vss-ask-video/SKILL.md works too.

Using another agent? Its personal skills folder
Codex
~/.agents/skills/
Cursor
~/.cursor/skills/
GitHub Copilot
~/.copilot/skills/
Gemini CLI
~/.gemini/skills/
Windsurf
~/.codeium/windsurf/skills/
OpenCode
~/.config/opencode/skills/

About

  • Asks the VSS agent's video_understanding tool a fresh visual question about a recorded clip
  • Explicitly declines when prior tool output, search hits or metadata already answer it
  • Probes the VSS agent and offers to deploy the base or lvs profile if none is running
  • Requires listing VST sensors before any generate call, even when the sensor is named
  • Handles follow-up detail questions after a coarse summary or a generated report

NVIDIA's Video Search and Summarization (VSS) blueprint runs a vision-language model over recorded footage so you can ask about it in words. This official skill is the part of that blueprint that answers a fresh visual question — what is happening in a clip, which objects or people appear, what colour something is, when an event occurs, whether a safety condition is met.

The interesting part is when it refuses

Most of the skill is a discipline about not running the VLM. Inference over video frames is expensive, so the skill will not fire when a database, an Elasticsearch hit, an MCP result, a prior summary or even a filename already answers the question — only when the agent genuinely needs a new look at the pixels, or when you explicitly want an answer verified against the footage. That constraint is what keeps a chatty agent from re-watching the same clip five times to answer five follow-ups.

Preflight

It checks its own dependencies before asking anything. It probes the VSS agent at http://${HOST_IP}:8000/docs, and if nothing is running it offers to hand off to the vss-deploy-profile skill to bring up the base profile (recommended for per-clip Q&A) or lvs. It then requires listing VST sensors before any /generate call — even when you have named the sensor yourself, because an asserted sensor name that does not exist produces a confidently wrong answer rather than an error.

Prerequisites

A running VSS profile that serves the video_understanding tool, and network reach to the VSS agent. The skill is versioned alongside the blueprint (3.2.0 at the time of writing) and lives in NVIDIA's agent-skills catalogue under Apache-2.0.

Try it

Once installed, ask your agent something like:

Watch the clip from camera 3 between 14:02 and 14:05 and tell me whether anyone entered without a hard hat.

SKILL.md

The instructions your agent loads, as published by NVIDIA.

SKILL.md
name
vss-ask-video
description
Use this skill to ask the VSS agent's video_understanding tool a fresh visual question about a recorded clip. Not for prior tool output, search hits, or metadata-answerable questions.
license
Apache-2.0
metadata
version: "3.2.0" github-url: "https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization" tags: "nvidia blueprint operational"

Video QnA using VLM through VSS Agent

Use this skill when you need details about the video which requires VLM to look at the video frames — for example the agent has no usable prior answer and needs a fresh look at the pixels for a specific clip.

---

When to Use
  • The user asks what happens in the video, what objects / people / actions appear, colors, timing, safety, or other visual facts that require watching the clip.
  • The user asks for details that cannot be answered from existing messages, summaries, Elasticsearch/MCP results, or filenames alone—you need model inference on the video.
  • Follow-up questions about content details after a coarse summary or after report generation.

Do not use this skill when a database / MCP / prior tool output already answers the question, unless the user explicitly wants verification against the video.

---

Deployment prerequisite

This skill requires a VSS profile that serves the video_understanding tool — typically base (recommended) or lvs. Before any request:

  1. Probe the VSS agent:
   curl -sf --max-time 5 "http://${HOST_IP}:8000/docs" >/dev/null
  1. If the probe fails, ask the user:

> "No VSS profile is running on `$HOST_IP`. Shall I deploy `base` (recommended for per-clip VLM QnA) using the `/vss-deploy-profile` skill? If you prefer `lvs`, say so."

  • If yes → hand off to /vss-deploy-profile -p base (or -p lvs if the user prefers). Return here once it succeeds.
  • If no → stop.
  1. If the probe passes, proceed.

---

Sensor prerequisite

You MUST list VST sensors before any `/generate` call. This is required even when the user names the sensor explicitly, even when the user asserts the video is already uploaded, and even when a previous turn appeared to use the same video. Do not skip this step.

  1. List sensors:
   curl -sf --max-time 5 "http://${HOST_IP}:30888/vst/api/v1/sensor/list" | jq '.[].name'
  1. Compare the returned name values against the user-supplied <sensor-id> (or filename stem, e.g. warehouse_safety_0001).
  1. If a matching sensor is present → proceed to the Agent workflow below.
  1. If no matching sensor is present — upload the video first, then re-list to confirm the new sensor appears:
   # filename: must not contain whitespace
   # timestamp: ISO 8601 UTC — default 2025-01-01T00:00:00.000Z if user did not specify
   curl -s -X PUT "http://${HOST_IP}:30888/vst/api/v1/storage/file/<filename>?timestamp=<timestamp>" \
     -H "Content-Type: application/octet-stream" \
     -H "Content-Length: <file_size_in_bytes>" \
     --upload-file /path/to/<filename> | jq .

See /vss-manage-video-io-storage for full upload semantics (v1 vs v2, conflict handling, delete flow). In interactive runs, confirm with the user before uploading. Never issue an unconditional PUT without first running the sensor-list check above — that is exactly the failure mode this prerequisite exists to prevent.

---

Agent workflow

The Sensor prerequisite above must have already confirmed (or made) the sensor exist on VST. Then:

  1. Clip — Identify sensor id, filename, or URL for one video segment. If ambiguous, ask the user.
  2. Call vss agent with the sensor id and ask for it to call video_understanding tool to answer the user's question.
  3. Return the vss agent's answer back to the user.
Query VSS agent (/generate)
# Set from deployment (compose / .env / host where vss-agent listens)
export VSS_AGENT_BASE_URL="http://localhost:8000"

curl -s -X POST "${VSS_AGENT_BASE_URL}/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message": "Call video_understanding tool to answer the following question about <sensor-id>: <user query>"}' | jq .
Response contract and extraction

/generate returns a JSON object with the assistant output in value, for example:

{"value":"<agent-think><agent-think-step ...>...</agent-think-step></agent-think>\n\n<final answer>\n\n"}

There is no separate clean-answer field. The consumable answer is the text in .value after removing any <agent-think>...</agent-think> block.

Required handling for this skill (and any downstream caller):

  1. Read .value from the JSON response.
  2. Strip <agent-think>...</agent-think> sections wherever they appear.
  3. Return only the remaining final-answer text to the user.

Example extraction:

curl -s -X POST "${VSS_AGENT_BASE_URL}/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message":"Call video_understanding tool to answer the following question about <sensor-id>: <user query>"}' \
| jq -r '.value' \
| python3 -c 'import re,sys; t=sys.stdin.read(); t=re.sub(r"<agent-think>.*?</agent-think>\s*", "", t, flags=re.S); print(t.strip())'

---

Cross-Reference
  • vss-manage-video-io-storage — VST storage/replay URLs so `VIDEO_URL` is valid for the VLM.
  • vss-generate-video-report — timestamped reports via Mode A (direct VLM) or Mode B (video-analytics incidents); this skill is VSS-agent `/generate` for ad-hoc video Q&A.
Install

Trains, distils, quantises, evaluates, exports and runs inference for RT-DETR real-time object detection in NVIDIA TAO, routing training through AutoML hyperparameter search by default.

Runs NVIDIA's NV-Reason-CXR-3B chest X-ray reasoning model through a documented wrapper, locally on GPU or via the public Hugging Face Space, and returns the model's full reasoning trace as engineering evidence.

Browse all skills →