The KAGAMI mark КАГАМИ
kagami.bg/academy · lesson · machine-readable viewUPDATED 2026-10-03
IDENTITY
module
GX10-04-78 · Active Speaker Detection NIM: who is speaking in a video
series
GX10 (local AI server class: NVIDIA GB10, e.g. ASUS Ascent GX10 / DGX Spark)
level
Advanced
duration
about 2 h
prerequisites
A machine with Docker and the NVIDIA Container Toolkit; an NGC personal API key with the NGC Catalog service; Python and pip for the sample client; ffmpeg; an MP4 (H.264) video, an audio source and a diarization JSON; a lawful basis and consent for any video that shows real people
trust_label
UPDATED 2026-10-03 (re-checked against the NIM documentation, pages last updated 2026-07-15 and 2026-09-30, and the build.nvidia.com catalogue page) · NOT TESTED: no GB10 machine was available and no command was run; GB10 is not named in the support matrix; whether the image runs on arm64 and on GB10 video decode hardware was not verified; no speed was measured
language
human view: bg · english edition: /en/academy/gx10/ (same file name)
previous / next
04-77_Content_Safety_Multimodal.html / 04-79_LipSync_Video.html
PURPOSE

Explain, strictly as documented, how the Active Speaker Detection NIM combines video, audio and diarization data to tell, per frame, which visible face is speaking: inputs, container launch, health check, sample gRPC client, output fields, input modes, environment variables, support matrix and licences, plus the data-protection duties that apply whenever a video shows real people. It makes no claim about GB10 compatibility or speed.

KEY CONCEPTS
COMMANDS / PATHS
CHECKLIST
NEXT MODULE

Series index: kagami.bg/academy/gx10/ · previous: 04-77 Nemotron Content Safety · next: 04-79 LipSync · related: 04-104 Riva Speech API (diarized transcripts), 04-126 Access control and GDPR · offer: Quick experiment (kagami.bg/stalbata/)

SOURCES
TAGS
gx10active-speaker-detectiondiarizationnimgrpcgdprvideo
UPDATED · 03.10.2026

Active Speaker Detection on GX10: Who Is Speaking in the Video

NVIDIA's Active Speaker Detection (ASD) tells, for each frame of a video, which of the visible faces is speaking. It combines the picture with the sound and with "diarization" — a list of who spoke when. It runs as a service in a container (gRPC). Video of people is personal data, so the lesson starts with the rules. We have not run any of this ourselves, and GB10 is not among the GPUs listed in the documentation.

⏱ 2 h Advanced GX10 ASD NIM · gRPC · ffmpeg Video of people · GDPR
Active Speaker Detection NIM (container)🔒 local Sample client (Python, gRPC)🔒 local Trial service in NVIDIA's cloud🌐 global
🔄
UPDATED · 03.10.2026 — what
The lesson was rebuilt from NVIDIA's official documentation (pages updated 15 July 2026 and 30 September 2026). We removed the code that sent requests to an address /v1/asd/predict with images and spectrograms in base64: the documentation has no such address, the service is gRPC and has a ready client. We also removed the table of "measured" times, standalone face detection with YOLO, automatic Whisper transcription as part of the service, and the examples about "drones" and "surveillance". We corrected: the input is video, audio and diarization (not just video and audio), and the result is per frame with a box, a face identifier and whether it is speaking. Added: formats, working modes, settings, the support table, the licences and — most important — the personal-data rules.
⚠️
What we have not run ourselves
We had no machine of the GB10 class, so there is no "TESTED" label. The commands are copied from the documentation and have not been run. GB10 is not in the support table, and the documentation does not say whether the image is built for ARM64; the service uses video-decode hardware (NVDEC) and we did not check whether GB10 has it. We have not measured speed, memory or accuracy. We also do not know how it behaves with Bulgarian speech.

01What you will learn

02Before you start

⚖️
Video of people and voices: the GDPR framework (not legal advice)
A person's image and voice are personal data (Regulation (EU) 2016/679, Art. 4(1)). Detecting faces and assigning each of them an identifier over time is processing of such data. If the service is used to recognise a specific person by their face, that is biometric data for unique identification — a special category under Art. 9, which is as a rule prohibited without an explicit ground. You need: a lawful basis (Art. 6), information to the people (Art. 13), for systematic monitoring a data protection impact assessment (Art. 35), a retention period, restricted access and a procedure for people's rights. Recording and analysing people without their knowledge is not permissible. For work meetings, get explicit consent from the participants before you start a recording; labour-law questions (for example with employees) should be settled with a lawyer. Recommendation: before you use real recordings, ask a lawyer or the data protection officer and write down their decisions. The AI Act (2024/1689) also restricts some uses — for example inferring emotions from faces in the workplace and in education. We have not checked for later amendments.

What the documentation says (as of 03.10.2026)

TopicWhat it says
What it doesDetects and identifies active speakers in a video by analysing the picture and diarized audio; it can track several speakers across different scenes
How it is builtVideo decoding on the GPU (NVDEC), audio aligned to frames, a diarization timeline and inference through the NVIDIA AR SDK on Triton
Image and portsnvcr.io/nim/nvidia/active-speaker-detection; gRPC on port 8001 by default; HTTP port 8000 with service addresses /v1/license, /v1/metadata, /v1/metrics
HardwareGPUs with Tensor cores of the Blackwell, Hopper, Ada, Ampere and Turing generations and with NVDEC. GB10 is not listed; for ARM64 the documentation says nothing
SoftwareDriver 571.21+; CUDA 12.8.1, TensorRT 10.9.0.34, Triton 2.56.0, DeepStream 8.0
SpeedThe first run includes model loading and does not show the real speed. There are no data for GB10 and we have not measured

Licences (per the official pages, 03.10.2026)

PartLicence per the pageWhat to watch
ModelNVIDIA Open Model License; additional note: MIT for the LR-ASD basisRead the licence for your use
Container (NIM)The "Governing Terms" page in the documentationWe did not check it in detail; read it before use
Trial service on build.nvidia.comNVIDIA API Trial Terms of ServiceThe video is sent to NVIDIA's cloud. Do not upload recordings of people there
Sample clients (nim-clients)We did not checkRead the licence file in the repository
⚠️
This is not legal advice
The table points to what the official pages say, and that may change. For a public or paid product, read the texts themselves or ask a lawyer.

03Steps

  1. What the service receives and returns

    ASD answers the question "which face in this frame is speaking?". A picture alone is not enough: it also needs sound and diarization — a list by times of which speaker said which word. ASD does not make the diarization; it comes from outside, for example from speech recognition (see the Riva lesson).

    For each frame the service returns a list of speakers with: a box around the face (speaker_bbox), the speaker number from the diarization (diarized_speaker_id), the face number (face_id), whether it is speaking at the moment (is_speaking) and how sure it is that this is a face (face_detection_confidence, from 0 to 1). The result is the model's guess, not a fact: a person checks it.

  2. Prepare the input

    The video is MP4 with H.264. For best performance make it "streamable" — the metadata moves to the start, and the service begins before it has received the whole file:

    bash
    ffmpeg -i clip.mp4 -movflags +faststart clip_streamable.mp4

    The audio is a separate file (WAV, MP3 or Opus) or the one embedded in the video is used (option --skip-audio). The diarization is JSON in the format of NVIDIA's speech recognition (Riva) or in the simple sample format of the documentation — an object with a words array, and in each element text, start, end (seconds) and speaker_id:

    json · sample diarization (invented text)
    {
      "language_code": "eng",
      "text": "Welcome to the meeting.",
      "words": [
        {"text": "Welcome", "start": 0.22, "end": 0.62, "speaker_id": "speaker_0"},
        {"text": "to",      "start": 0.62, "end": 0.80, "speaker_id": "speaker_0"}
      ]
    }
    ✅
    In streaming mode the diarization must be ready from the first request
    If it is not provided with the first request, the session is treated as one without diarization and results can be wrong. Provide it in advance or interleave it with the stream.
  3. Log in to the catalogue and start the service

    Create a personal NGC key with "NGC Catalog" and log in. The key is a password: pass it through an environment variable.

    bash
    export NGC_API_KEY=<your-key>
    echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

    The user name $oauthtoken is literally so. The start command is from the documentation; we only bound the ports to the local address so they are not visible from outside (the documentation has -p 8000:8000 and -p 8001:8001):

    bash
    docker run -it --rm --name=active-speaker-detection-nim \
      --runtime=nvidia \
      --gpus all \
      --shm-size=8GB \
      -e NGC_API_KEY=$NGC_API_KEY \
      -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1 \
      -e NIM_HTTP_API_PORT=8000 \
      -e NIM_GRPC_API_PORT=8001 \
      -p 127.0.0.1:8000:8000 \
      -p 127.0.0.1:8001:8001 \
      nvcr.io/nim/nvidia/active-speaker-detection:latest

    We have not run this command. If the service does not start for you, the machine itself may be the reason: the documentation does not list GB10 and does not say whether the image is built for ARM64.

    • Profile: if you do not set NIM_MODEL_PROFILE, the service chooses by itself according to the card. Set for a different card, it leads to an error at run time.
    • Speakers: by default up to 5 audio streams (NV_AI4M_ASD_MAX_SPEAKERS).
    • Parallel requests: 1 per card by default; more means more memory and slower for each.
    • Protection: by default the connection is not encrypted; the documentation has modes tls and mtls (NIM_SSL_MODE). Keep the ports on the local address.
  4. Check that the service is ready

    Health is checked over gRPC with grpcurl. The documentation shows a package for amd64; on ARM64 take the version for your architecture from the project page.

    bash
    wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
    grpcurl --plaintext --proto health.proto localhost:8001 grpc.health.v1.Health/Check

    When ready, the answer is { "status": "SERVING" }.

  5. Run the sample client

    bash
    git clone https://github.com/NVIDIA-Maxine/nim-clients.git
    cd nim-clients/active-speaker-detection
    pip install -r requirements.txt
    cd scripts
    
    python active_speaker_detection.py \
      --target 127.0.0.1:8001 \
      --video-input clip_streamable.mp4 \
      --audio-input speech.wav \
      --diarization-input diarization.json \
      --output result.mp4

    All parameters are optional; without them the client uses sample files from the repository. For embedded audio add --skip-audio and drop --audio-input. The client draws boxes on the output video: green — the face is speaking; blue — has an assigned audio track but is not speaking at the moment; red — a tracked face without an audio track. The first run does not show the real speed — it includes loading the models. We have not run this command.

  6. Settings for a better result

    SettingWhat it does (per the documentation)
    NV_AI4M_ASD_VA_SMOOTHINGSmoothing of the "speaking / not speaking" decisions from the diarization: off, low or high (default)
    NV_AI4M_ASD_MULTIVIEWA model for linking the same people across several simultaneous streams; off by default
    NV_AI4M_ASD_*_TIMEOUT_SHow long the service waits for video, audio or diarization for a frame; 30 seconds by default. For the diarization there is a mode for lateness: adaptive, single_speaker or terminate

    The documentation also has sections on several GPUs and on monitoring the service (metrics). We skip them here because they are not about a single machine.

  7. What to do with the result — and what not to

    • Use it to help a person (for example to mark who is speaking in a recording of a meeting the participants have approved), not to make decisions about people without human review.
    • Keep the minimum: the output boxes per frame are enough; do not keep faces and video longer than needed and set a deletion period.
    • Restrict who has access to the recordings and results; log who viewed them.
    • Do not link face_id to people's names unless you have a ground and consent and have discussed it with a lawyer — that is already identification.
    • Tell people that they are being recorded and analysed before you start.

04Check

Quiz

1. Which three inputs does ASD use?

2. How does the client talk to the service?

3. What does it mean for video of people that the trial service on build.nvidia.com is "in the cloud"?

4. What is face_id?

05What next

06Sources

  1. Active Speaker Detection NIM — documentation 🔒 local — overview, launch, usage, support table.
  2. Active Speaker Detection — NVIDIA catalogue page 🌐 global — description, model licence, trial-service terms.
  3. NVIDIA-Maxine/nim-clients — sample clients.
  4. NVIDIA Riva: speech recognition — how diarization is obtained.
  5. EUR-Lex: Regulation (EU) 2016/679 (GDPR) — Art. 4, 6, 9, 13 and 35.
  6. EUR-Lex: Regulation (EU) 2024/1689 (AI Act) — Art. 5.