TESTED · 28.10.2026

Bulgarian speech to text: trialing two local models

A new model promises better speech recognition. How do you find out whether it is better for Bulgarian, not just in the announcement? The same audio, a reference text and one yardstick. We trialed Nemotron 3.5 Transcribe against Whisper large-v3 on our own machine — here is the method and what came out.

⏱ ~18 min Intermediate KAGAMI Way · I33 test · word errors · translation

Whisper large-v3🔒 local Nemotron 3.5 Transcribe🔒 local FLEURS — test recordings🌐 public data

01What you will learn

How to compare speech-to-text models fairly: a test with reference text and a word error rate.
Why “normalising” the text decides half of the comparison.
How to instruct a model that takes directions in words — language, verbatim or clean, names.
What the percentage does not tell you — and why a person reads the errors.

02Before you start

A computer where the models run with you 🔒 local — the voice never leaves. Whisper also runs on a CPU only; Nemotron needs a graphics card with a lot of memory.
Memory for Nemotron: the full weights are about 63 GB. We loaded them “lightened” (FP8 at load time) — about half. Our machine has 128 GB of unified memory.
Python and ffmpeg — to prepare the audio and do the maths.
About an hour and a half to download the model on an ordinary connection.

⛔

Keep a memory reserve

While loading, a large model briefly takes more than you allowed it. If other services run on the same machine, check free memory before starting and keep at least 20 GB in reserve. Stop the model after the batch.

03Steps

Get recordings with reference text

Without reference text you cannot tell who is wrong — only who writes differently. So start with a public test: Google's FLEURS has Bulgarian recordings and their text. We took 60 different sentences, one recording per sentence, with a fixed random choice — so the trial can be repeated.
Audio in the format the models expect
```
ffmpeg -i recording.mp3 -ac 1 -ar 16000 recording.wav
```

Measure with one yardstick

The yardstick is the word error rate: how many words are substituted, dropped or added compared with the reference. Before scoring, both texts are “normalised” the same way — lower case, no punctuation. Otherwise a model loses points for a comma, not for a word.

Word error rate · Python

import re, unicodedata

def words(s):
    s = unicodedata.normalize("NFC", s.lower()).replace("ѝ", "и")
    return re.sub(r"[^\w\s]", " ", s).split()

def distance(a, b):  # words to substitute, delete or insert
    d = list(range(len(b) + 1))
    for i, x in enumerate(a, 1):
        p, d[0] = d[0], i
        for j, y in enumerate(b, 1):
            p, d[j] = d[j], min(d[j] + 1, d[j - 1] + 1, p + (x != y))
    return d[-1]

def wer(refs, hyps):
    e = sum(distance(words(r), words(h)) for r, h in zip(refs, hyps))
    return 100 * e / sum(len(words(r)) for r in refs)

⚠️

Trap: numbers

We do not normalise numbers. “M16” against “M sixteen” counts as an error. That is why figures from different sites cannot be compared with each other — everyone normalises their own way.

Run both models under the same conditions

Tell both the language explicitly and set “temperature” to 0 — no randomness. Whisper takes the language as a setting. Nemotron takes an instruction in words together with the audio: language, verbatim or clean, how to write numbers, which names to expect.

Instruction to Nemotron · transcription

Audio transcription. Language: Bulgarian, in its standard writing system.
Formatting: punctuation and capitalization. Output: transcript only.

Request to the local server (vLLM) · Python

import base64, requests

audio = base64.b64encode(open("recording.wav", "rb").read()).decode()
r = requests.post("http://localhost:<port>/v1/chat/completions", json={
    "model": "nemotron", "temperature": 0,
    "chat_template_kwargs": {"enable_thinking": False},
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": "Audio transcription. Language: Bulgarian. Output: transcript only."},
        {"type": "audio_url", "audio_url": {"url": "data:audio/wav;base64," + audio}}]}]})
print(r.json()["choices"][0]["message"]["content"])

💡

Names and terms

Add “Context terms:” to the instruction and list the names that will be heard — organisations, people, places. The model then spells them correctly instead of guessing.

Read the numbers — and the errors

On the 60 Bulgarian sentences:
Result · 28.10.2026
```
Model                         Word errors   Better in
Whisper large-v3              15.23 %       14 sentences
Nemotron 3.5 Transcribe       11.86 %       36 sentences
(equal: 10 sentences)
```
Nemotron writes Bulgarian endings and compound words more cleanly where Whisper misspells them. But read the opposite cases too. In one sentence Nemotron “completed the meaning” on its own and wrote something fluent but wrong — a rifle became “descending from M sixteen”. In a live recording it once flipped the meaning of a whole sentence.

✅
The rule
Fewer word errors does not mean safer text. Minutes, a quote or a decision written by a model are read by a person against the audio.

⚠️
Trap: speed
Our Whisper ran on the CPU, Nemotron on the graphics card. So 17 seconds against 1 second per recording compares the machines, not the models. Compare speed only on the same hardware.
Try a live recording and translation

Test recordings are read aloud. Live speech in a hall is harder. We ran 3 minutes of a live Bulgarian lecture through both — the two texts differed in about 12% of words, with no reference to say who was right. Here a person who listens decides.

Nemotron can also translate from audio: we ran an English talk with the instruction “Please translate the given speech to Bulgarian.” and got fluent Bulgarian text. Whisper only translates into English.

⚠️
Trap: the wrong language
Tell Whisper “Bulgarian” when the recording is in English and it returns a half-translation full of errors. First check the language of the recording — Whisper can detect it if you do not set one.