The KAGAMI mark КАГАМИ
kagami.bg/academy · lesson · machine-readable viewUPDATED 2026-10-03
IDENTITY
module
GX10-04-100 · Experiment tracking with Weights & Biases
series
GX10 (local AI server class: NVIDIA GB10, e.g. ASUS Ascent GX10 / DGX Spark)
level
Intermediate
duration
45 min
prerequisites
Python 3 with pip, a Weights & Biases account and API key, internet access (or offline mode)
trust_label
UPDATED 2026-10-03 (compared with the current W&B quickstart, sweeps tutorial and environment-variable reference) · NOT TESTED (no GB10 machine available) · commands not run
status_note
NVIDIA marks the "AI Observability for Data Flywheel" blueprint (W&B partner blueprint) as deprecated (checked 2026-10-03). This lesson teaches the W&B SDK itself, not that blueprint
language
human view: bg · english edition: /en/academy/gx10/ (same file name)
previous / next
GX10 series index / 04-101_Safety_Agentic_AI.html
PURPOSE

Record the runs of a model-training or evaluation script (hyperparameters, metrics) in Weights & Biases, compare them, work offline when needed, and automate a hyperparameter search with a sweep.

KEY CONCEPTS
COMMANDS / PATHS
CHECKLIST
NEXT MODULE

04-101 · Safety for agentic AI (04-101_Safety_Agentic_AI.html) · series index: kagami.bg/academy/gx10/ · offer: Quick experiment (kagami.bg/stalbata/)

SOURCES
TAGS
gx10nvidia-gb10wandbexperiment-trackingsweepsobservabilitypython
UPDATED · 03.10.2026

W&B on GX10: Tracking ML Experiments

When you train or evaluate a model, after a dozen attempts you no longer remember which settings gave which result. Weights & Biases records every attempt (run) with its settings and metrics so you can compare them. Here we connect it to a script on a GX10, work without a network too, and start an automatic search for settings.

⏱ 45 min Intermediate GX10 Python · W&B · sweeps
Python · wandb (the library)🔒 local W&B (service and dashboard)🌐 global
🔄
UPDATED · 03.10.2026 — what changed
We reworked the lesson against the current W&B documentation (quickstart, sweeps tutorial, environment variables). We fixed the code: in the old version the variable config was not defined and wandb.agent.run(...) is not part of the interface — it is now wandb.agent(...). We removed what we could not confirm: Slack alerts, a model registry, a LLaMA Factory integration and the automatic "data flywheel" — they were not checked, and NVIDIA's blueprint for it is deprecated. We added: where the data goes (cloud versus offline), how to keep the key safe and why a sweep and its runs must share a project.
⚠️
NVIDIA's blueprint is deprecated
NVIDIA's page "AI Observability for Data Flywheel" (a partner blueprint with W&B) says as of 03.10.2026 "This Blueprint has been deprecated" and recommends moving to another solution. This lesson does not use it: it teaches the W&B library itself. If you need a full "data flywheel", do not start from this blueprint.
⚠️
What we have not run ourselves
We had no GB10-class machine. The code was compared with the W&B documentation but we have not run it — which is why there is no "TESTED" label. We did not check how wandb behaves on Arm64 or how offline work is synchronised.

01What you'll learn

02Before you start

🌐
Where the data goes
By default the metrics, settings and all logged files are sent to the W&B service — it is global, not local. Do not log personal data, client texts or secrets to it. For work without sending there is an offline mode (step 5). Self-hosting (wandb/local) is mentioned in the environment-variable documentation; we do not cover it in this lesson.

03Steps

  1. Create an API key

    Log in to the dashboard, open your profile → User settings → API keys → New key, give it a name and copy the key at once. Why at once: the full key is shown only once; afterwards you see only its beginning and if you lose it you create a new one.

    ✅
    The key is a secret
    Do not write it in the script, in Git or on a page. Keep it in an environment variable or a password manager. If it leaks — delete it in the dashboard and create a new one.
  2. Install and log in

    A virtual environment keeps the project's packages separate from the system.

    bash
    python3 -m venv .venv
    . .venv/bin/activate
    pip install wandb
    wandb login   # asks for the key

    In automated environments the key is passed with export WANDB_API_KEY=<key> before the script (the variable is described in the documentation). ⚠️ We did not check the installation on Arm64.

  3. The first run

    wandb.init() starts a run: you give a project and a dictionary of settings. Everything you record with run.log() becomes a chart in the dashboard. The example is from the W&B quickstart and simulates training:

    python · first_run.py
    import wandb
    import random
    
    wandb.login()
    
    project = "my-awesome-project"
    config = {"epochs": 10, "lr": 0.01}
    
    with wandb.init(project=project, config=config) as run:
        offset = random.random() / 5
        for epoch in range(2, config["epochs"]):
            acc = 1 - 2**-config["epochs"] - random.random() / config["epochs"] - offset
            loss = 2**-config["epochs"] + random.random() / config["epochs"] + offset
            run.log({"accuracy": acc, "loss": loss})

    Run it a few times and the dashboard will show several runs with random names. In your own script put the same run.log({...}) where you get the loss at each step.

  4. Compare runs

    In the dashboard open the project: the "Runs" column lists them and the charts overlay the metrics. The point of the settings in config is here — you can see which lr gave a lower loss. The documentation also has a parallel-coordinates chart and a parameter-importance analysis (⚠️ not tried by us).

  5. Working without sending: offline mode

    If the script must not reach the internet or the data must not leave, set the variable before running:

    bash
    export WANDB_MODE=offline   # keeps the metadata locally, no sync
    python first_run.py
    # WANDB_MODE=disabled turns W&B off completely

    How offline runs are uploaded later is described in the W&B documentation; we did not try it, so we give no command.

  6. Sweep: automatic search for settings

    A sweep tries different values of the settings and records a run for each. You describe the search space, register the sweep and start an "agent" that calls your function. The example is from the W&B tutorial:

    python · sweep.py
    import wandb
    
    def objective(config):
        score = config.x**3 + config.y
        return score
    
    def main():
        with wandb.init(project="my-first-sweep") as run:
            score = objective(run.config)
            run.log({"score": score})
    
    sweep_configuration = {
        "method": "random",
        "metric": {"goal": "minimize", "name": "score"},
        "parameters": {
            "x": {"max": 0.1, "min": 0.01},
            "y": {"values": [1, 3, 7]},
        },
    }
    
    sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")
    wandb.agent(sweep_id, function=main, count=10)
    ⚠️
    One project for the sweep and its runs
    The project name in wandb.init() must match the one in wandb.sweep(). If you use multiprocessing, wrap wandb.sweep() and wandb.agent() in if __name__ == "__main__":. The agent is stopped with Ctrl+C (a second time to end it).

    The method here is random. Other methods and options are in the "Define sweep configuration" section of the documentation (not opened by us).

  7. What else there is

    The quickstart also points to: tracking models and datasets with Artifacts, sharing in the Registry and reports. For applications with language models the same vendor has W&B Weave. None of these is run in this lesson.

04Check

Quiz

1. Which function starts a new run?

2. How do you make the script not send data to the cloud?

3. Where is the right place for the API key?

4. What applies to the project in a sweep?

05What's next

06Sources

  1. Weights & Biases: quickstart 🌐 global — key, installation, first run; checked 03.10.2026.
  2. W&B: sweeps tutorial — configuration, wandb.sweep, wandb.agent; checked 03.10.2026.
  3. W&B: environment variables — WANDB_API_KEY, WANDB_MODE, WANDB_BASE_URL.
  4. NVIDIA: AI Observability for Data Flywheel — page of the deprecated blueprint; repository.