Active Speaker Detection on GX10: Who Is Speaking in the Video
NVIDIA's Active Speaker Detection (ASD) tells, for each frame of a video, which of the visible faces is speaking. It combines the picture with the sound and with "diarization" — a list of who spoke when. It runs as a service in a container (gRPC). Video of people is personal data, so the lesson starts with the rules. We have not run any of this ourselves, and GB10 is not among the GPUs listed in the documentation.
/v1/asd/predict with images and spectrograms in base64: the documentation has no such address, the service is gRPC and has a ready client. We also removed the table of "measured" times, standalone face detection with YOLO, automatic Whisper transcription as part of the service, and the examples about "drones" and "surveillance". We corrected: the input is video, audio and diarization (not just video and audio), and the result is per frame with a box, a face identifier and whether it is speaking. Added: formats, working modes, settings, the support table, the licences and — most important — the personal-data rules.
01What you will learn
- What ASD does and why using it on video of people has rules.
- What the service needs: video, audio and diarization.
- How it is launched and checked per the documentation and how the sample client is run.
- What it returns for each frame and how to read the coloured boxes.
- Which settings and limits exist (number of speakers, modes, timeouts).
- How to protect the data: how much, for how long and where.
02Before you start
- A machine with Docker and the NVIDIA Container Toolkit; driver 571.21 or later (per the support table).
- A personal NGC key with the "NGC Catalog" service included (to pull the image). The key is a secret.
- Python and
pipfor the sample client,ffmpegand Git. - A video in MP4 with H.264; audio (WAV, MP3 or Opus) or audio embedded in the video; a diarization file.
- Only invented or your own recordings with consent for the trial: for example a recording of your own team that has agreed.
What the documentation says (as of 03.10.2026)
| Topic | What it says |
|---|---|
| What it does | Detects and identifies active speakers in a video by analysing the picture and diarized audio; it can track several speakers across different scenes |
| How it is built | Video decoding on the GPU (NVDEC), audio aligned to frames, a diarization timeline and inference through the NVIDIA AR SDK on Triton |
| Image and ports | nvcr.io/nim/nvidia/active-speaker-detection; gRPC on port 8001 by default; HTTP port 8000 with service addresses /v1/license, /v1/metadata, /v1/metrics |
| Hardware | GPUs with Tensor cores of the Blackwell, Hopper, Ada, Ampere and Turing generations and with NVDEC. GB10 is not listed; for ARM64 the documentation says nothing |
| Software | Driver 571.21+; CUDA 12.8.1, TensorRT 10.9.0.34, Triton 2.56.0, DeepStream 8.0 |
| Speed | The first run includes model loading and does not show the real speed. There are no data for GB10 and we have not measured |
Licences (per the official pages, 03.10.2026)
| Part | Licence per the page | What to watch |
|---|---|---|
| Model | NVIDIA Open Model License; additional note: MIT for the LR-ASD basis | Read the licence for your use |
| Container (NIM) | The "Governing Terms" page in the documentation | We did not check it in detail; read it before use |
| Trial service on build.nvidia.com | NVIDIA API Trial Terms of Service | The video is sent to NVIDIA's cloud. Do not upload recordings of people there |
| Sample clients (nim-clients) | We did not check | Read the licence file in the repository |
03Steps
-
What the service receives and returns
ASD answers the question "which face in this frame is speaking?". A picture alone is not enough: it also needs sound and diarization — a list by times of which speaker said which word. ASD does not make the diarization; it comes from outside, for example from speech recognition (see the Riva lesson).
For each frame the service returns a list of speakers with: a box around the face (
speaker_bbox), the speaker number from the diarization (diarized_speaker_id), the face number (face_id), whether it is speaking at the moment (is_speaking) and how sure it is that this is a face (face_detection_confidence, from 0 to 1). The result is the model's guess, not a fact: a person checks it. -
Prepare the input
The video is MP4 with H.264. For best performance make it "streamable" — the metadata moves to the start, and the service begins before it has received the whole file:
bashffmpeg -i clip.mp4 -movflags +faststart clip_streamable.mp4The audio is a separate file (WAV, MP3 or Opus) or the one embedded in the video is used (option
--skip-audio). The diarization is JSON in the format of NVIDIA's speech recognition (Riva) or in the simple sample format of the documentation — an object with awordsarray, and in each elementtext,start,end(seconds) andspeaker_id:json · sample diarization (invented text){ "language_code": "eng", "text": "Welcome to the meeting.", "words": [ {"text": "Welcome", "start": 0.22, "end": 0.62, "speaker_id": "speaker_0"}, {"text": "to", "start": 0.62, "end": 0.80, "speaker_id": "speaker_0"} ] }✅In streaming mode the diarization must be ready from the first requestIf it is not provided with the first request, the session is treated as one without diarization and results can be wrong. Provide it in advance or interleave it with the stream. -
Log in to the catalogue and start the service
Create a personal NGC key with "NGC Catalog" and log in. The key is a password: pass it through an environment variable.
bashexport NGC_API_KEY=<your-key> echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdinThe user name
$oauthtokenis literally so. The start command is from the documentation; we only bound the ports to the local address so they are not visible from outside (the documentation has-p 8000:8000and-p 8001:8001):bashdocker run -it --rm --name=active-speaker-detection-nim \ --runtime=nvidia \ --gpus all \ --shm-size=8GB \ -e NGC_API_KEY=$NGC_API_KEY \ -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1 \ -e NIM_HTTP_API_PORT=8000 \ -e NIM_GRPC_API_PORT=8001 \ -p 127.0.0.1:8000:8000 \ -p 127.0.0.1:8001:8001 \ nvcr.io/nim/nvidia/active-speaker-detection:latestWe have not run this command. If the service does not start for you, the machine itself may be the reason: the documentation does not list GB10 and does not say whether the image is built for ARM64.
- Profile: if you do not set
NIM_MODEL_PROFILE, the service chooses by itself according to the card. Set for a different card, it leads to an error at run time. - Speakers: by default up to 5 audio streams (
NV_AI4M_ASD_MAX_SPEAKERS). - Parallel requests: 1 per card by default; more means more memory and slower for each.
- Protection: by default the connection is not encrypted; the documentation has modes
tlsandmtls(NIM_SSL_MODE). Keep the ports on the local address.
- Profile: if you do not set
-
Check that the service is ready
Health is checked over gRPC with
grpcurl. The documentation shows a package for amd64; on ARM64 take the version for your architecture from the project page.bashwget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto grpcurl --plaintext --proto health.proto localhost:8001 grpc.health.v1.Health/CheckWhen ready, the answer is
{ "status": "SERVING" }. -
Run the sample client
bashgit clone https://github.com/NVIDIA-Maxine/nim-clients.git cd nim-clients/active-speaker-detection pip install -r requirements.txt cd scripts python active_speaker_detection.py \ --target 127.0.0.1:8001 \ --video-input clip_streamable.mp4 \ --audio-input speech.wav \ --diarization-input diarization.json \ --output result.mp4All parameters are optional; without them the client uses sample files from the repository. For embedded audio add
--skip-audioand drop--audio-input. The client draws boxes on the output video: green — the face is speaking; blue — has an assigned audio track but is not speaking at the moment; red — a tracked face without an audio track. The first run does not show the real speed — it includes loading the models. We have not run this command. -
Settings for a better result
Setting What it does (per the documentation) NV_AI4M_ASD_VA_SMOOTHINGSmoothing of the "speaking / not speaking" decisions from the diarization: off,loworhigh(default)NV_AI4M_ASD_MULTIVIEWA model for linking the same people across several simultaneous streams; off by default NV_AI4M_ASD_*_TIMEOUT_SHow long the service waits for video, audio or diarization for a frame; 30 seconds by default. For the diarization there is a mode for lateness: adaptive,single_speakerorterminateThe documentation also has sections on several GPUs and on monitoring the service (metrics). We skip them here because they are not about a single machine.
-
What to do with the result — and what not to
- Use it to help a person (for example to mark who is speaking in a recording of a meeting the participants have approved), not to make decisions about people without human review.
- Keep the minimum: the output boxes per frame are enough; do not keep faces and video longer than needed and set a deletion period.
- Restrict who has access to the recordings and results; log who viewed them.
- Do not link
face_idto people's names unless you have a ground and consent and have discussed it with a lawyer — that is already identification. - Tell people that they are being recorded and analysed before you start.
04Check
- You know the lesson was not run on GB10 and that support for GB10 and ARM64 is not confirmed.
- You have a ground, the consent of everyone shown and a retention period — before you use a real recording.
- You have not uploaded recordings of people to the trial service in the cloud.
- The ports are bound to the local address; the NGC key is not in a file that is shared.
- The video is MP4 with H.264 and streamable; the diarization is ready before the first request.
- The health check returns
SERVINGand the client writes the output file. - A person has reviewed the result; it is not used alone for decisions about people.
Quiz
1. Which three inputs does ASD use?
2. How does the client talk to the service?
3. What does it mean for video of people that the trial service on build.nvidia.com is "in the cloud"?
4. What is face_id?
05What next
06Sources
- Active Speaker Detection NIM — documentation 🔒 local — overview, launch, usage, support table.
- Active Speaker Detection — NVIDIA catalogue page 🌐 global — description, model licence, trial-service terms.
- NVIDIA-Maxine/nim-clients — sample clients.
- NVIDIA Riva: speech recognition — how diarization is obtained.
- EUR-Lex: Regulation (EU) 2016/679 (GDPR) — Art. 4, 6, 9, 13 and 35.
- EUR-Lex: Regulation (EU) 2024/1689 (AI Act) — Art. 5.