Video Agent with NVIDIA VSS on GX10
You upload a video recording, ask in plain language "is the worker wearing a hard hat?" and get an answer with a supporting frame — plus a ready report in Markdown and PDF. This is NVIDIA's ready-made video agent, brought up with one script, rather than one more home-built pipeline.
v3.2.1). The old version described a home-built pipeline with other models and another vector database, and a table of timings "on GB10" without a source. We removed: that pipeline, the performance table (numbers with no measurement), the scenarios with specific customers and sites, and the addresses and code with internal paths. We added: what the base agent consists of, what the documentation says about the GB10 class, the exact Docker and driver requirements, the human in the loop for reports, a security note (VSS is meant for an isolated network) and a legal framework for recordings of people.
01What you'll learn
- What NVIDIA's video agent consists of and what each part does.
- What the documentation requires of the machine and what it says specifically about the GB10 class.
- How to deploy the base agent, upload a recording and ask it questions.
- How to get a report, and why there is a step where a person approves or corrects the prompt sent to the model.
- How to keep it in an isolated network and what to settle with a lawyer before running it on recordings of people.
02Before you start
- A machine of the NVIDIA GB10 class with DGX OS and access to its terminal. For DGX Spark the documentation names DGX OS 7.4.0 and driver 580.95.05; the system minimum is 128 GB of memory and a 1 TB SSD.
- Docker 28.3.3 or newer but older than 29.5.0 (outside that band pulling some images fails), Docker Compose v2.39.1+, NVIDIA Container Toolkit 1.17.8+, NGC CLI 4.10.0+, Git with Git LFS. Docker from snap packages does not work — use the official repository.
- Keys: a personal API key from NGC (legacy keys do not work for the VLM image) and a read token from Hugging Face. Both are set only as environment variables in the terminal.
- Some free time for the first download — the images are large. How large exactly, we have not measured.
- Legal check: see the box in step 7. Before processing recordings of real people, talk to a lawyer.
03Steps
-
What you get
VSS is a set of reference architectures. It has three layers: real-time video intelligence (object detection, embeddings, scene descriptions from a vision-language model), downstream analytics (behaviour, alert verification) and an agent that assembles all of that into reports, answers and search. Here we deploy the smallest piece — the base agent:
Part Job VSS Agent Takes the request, decides which tools to call and assembles the answer. The recommended LLM is Nemotron Nano 9B v2 Agent UI Web chat with video upload and several views Video IO & Storage (VIOS) Ingests, records and plays back the videos for the agent LLM (NIM) Reasons and picks tools VLM — Cosmos3 Nano Reasoner (NIM) "Watches" the video and answers questions about it Phoenix Traces the agent's steps All parts can also be used on their own through REST and MCP; for this lesson we need only the chat.
-
Prepare the machine
First check the versions:
docker version,docker compose version,ngc --version. If Docker is newer than 29.5.0 or older than 28.3.3, the images may fail to download. Then apply the settings from the requirements page: a file in/etc/sysctl.dwith network parameters and a cache-cleaner script (for DGX Spark) that periodically drops the system cache, and have Docker use thecgroupfscgroup driver. Take the exact text of the scripts from the requirements page — it changes, and nothing here has been run.💡Why the cache cleanerThe memory of these machines is shared between the processor and the graphics chip; with large models the system cache can eat it up. That is why NVIDIA requires this script on DGX Spark. It works only while the session is alive — for permanence you make it a system service. -
Download the package and the sample recording
The deployment package is in an NVIDIA repository on GitHub and uses Git LFS for large files. The documentation points to the tag
v3.2.1.bash · not rungit clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git cd video-search-and-summarization git checkout tags/v3.2.1 git lfs install git lfs pullThe sample recording comes from NGC. It is needed only for a trial; you can also use your own short recording, provided it shows no people who have not given consent.
bash · not runngc registry resource download-version nvidia/vss-developer/dev-profile-sample-data:3.2.0Before that, set up the NGC CLI with
ngc config setand choose an organization — without one the resources cannot be downloaded. -
Set the keys and deploy
The keys are set as environment variables in the same terminal you run the script from. Do not write them to a file and do not commit them to a repository.
bash · not runexport NGC_CLI_API_KEY='<ngc-api-key>' export HF_TOKEN='<hf-token>' # all options of the script deploy/docker/scripts/dev-profile.sh --helpFor DGX Spark the documentation shows two ways. The first — with an external LLM (its address is yours: another machine, or a local server on another port):
bash · not runexport LLM_ENDPOINT_URL=https://<llm-address> deploy/docker/scripts/dev-profile.sh up -p base -H DGX-SPARK --use-remote-llmThe second — both the LLM and the VLM as containers on the same machine (according to the "edge" deployment guide):
bash · not rundeploy/docker/scripts/dev-profile.sh up -p base \ --hardware-profile DGX-SPARK \ --llm nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 \ --vlm nvidia/cosmos3-nano-reasoner⚠️The two examples are not written the same wayThe first uses-H, the second--hardware-profile. We have not checked whether they are the same; run--helpand use the form your version shows. The "edge" guide also has a variant with a smaller LLM (Nemotron 3 Nano 4B) with its own configuration file — we do not cover it here.The script downloads the containers and starts them. When it finishes, check:
bash · not rundocker ps curl -fsS http://<server-address>:8000/healthThe agent should be reachable on port
7777(chat);8000is its REST interface and30888the video interface. If the page does not open, the documentation points to checking the machine's firewall: containers on a bridge network may not reach services on the host. -
Upload a recording and ask
Open the chat on port 7777 from a browser in the same network. Drag the video into the chat window — the agent answers with playback. Then you ask, for example (the examples are from the documentation, for the sample recording):
chatIs the worker wearing PPE? When did the worker climb up the ladder? List all available sensors Take a snapshot of <sensor-id> at <timestamp>The agent shows its intermediate steps before giving the answer. The documentation gives the examples in English; how it behaves in Bulgarian, we have not checked. The answer is the model's opinion, not evidence: for every important claim, look at the frame and the recording.
-
A report — and the human in the loop
Ask "Can you generate a report for this video?". Before it starts, the agent shows you the prompt for the VLM and waits. You have five options:
- submit empty — you approve and a report is generated;
- type a new prompt or edit the one shown;
/generate <description>— the LLM drafts a prompt from your description;/refine <instructions>— the LLM adjusts the current one;/cancel— you decline.
This is a good practice, because the report is only as accurate as the prompt. Reports come in Markdown and PDF and sit on port 8000 under
static. By default they are kept in memory — after a restart of the agent container they disappear. For permanent storage you have to turn on saving a local copy and mount a directory from the machine (described in the documentation for reports). -
People in the frame: the legal framework
⚖️Recording and analysing people has legal requirementsVideo surveillance, including automatic description of recordings with people and the reports made from them, is processing of personal data and falls under data-protection rules (GDPR) and national law. The legal basis, informing people, retention period, access to the recordings and reports, and the rights of the people concerned are questions that depend on the site and the purpose. Recommendation: before you run a system on real recordings, check with a lawyer and write down their decisions. This lesson is technical and is not legal advice.- Good technical practice: start with sample recordings without people; limit who has access to the chat and the reports; do not turn on recognition of specific individuals.
- Keep recordings and reports as long as needed and no longer; set a period and apply it.
- Do not expose the ports to the internet — the agent has no login of its own.
-
Shut it down
bash · not rundeploy/docker/scripts/dev-profile.sh downThe command stops and removes the agent's containers. When switching profile (for example to summarization), the documentation advises clearing the old data volumes as well.
04Check
- The versions of Docker, Compose, Container Toolkit and NGC CLI meet the requirements, and Docker is older than 29.5.0.
- The kernel settings and the cache cleaner are applied as the requirements page describes.
- The keys are set as environment variables, not in a file.
- The package is at the documented tag and the files pulled through Git LFS are downloaded.
- The containers are healthy in
docker psand the health endpoint answers. - You uploaded a sample recording and got an answer, including to a follow-up question.
- You generated a report and used the prompt-review step at least once.
- The ports are not visible outside the trusted network.
- Before pointing the system at recordings of real people, you talked to a lawyer and wrote down the period and access.
Quiz
1. What kind of network does NVIDIA expect for a VSS deployment?
2. Which profiles are listed for DGX Spark in the requirements table?
3. Why does the agent show you the VLM prompt before it starts a report?
4. What is the most sensible thing to do before running the agent on recordings of real people?
05What's next
06Sources
- NVIDIA VSS · Introduction 🔒 local — architecture, three layers, isolated network.
- NVIDIA VSS · Prerequisites — hardware, versions, kernel settings, keys, GPU profiles.
- NVIDIA VSS · Quickstart — deploying the base agent, questions, report, shutdown.
- NVIDIA VSS · Edge Deployment — DGX Spark and local models.
- GitHub: NVIDIA-AI-Blueprints/video-search-and-summarization — the deployment package.
- build.nvidia.com · VSS Blueprint 🌐 global — project page.