The KAGAMI mark КАГАМИ
kagami.bg/academy · lesson · machine-readable viewTESTED 2026-09-23 · VERIFIED 2026-10-01 · UPDATED 2026-10-01
IDENTITY
module
KW I28 · 10 models in Bulgarian: what survived
series
KAGAMI Way · Track I — Infrastructure
level
Intermediate
duration
~22 min
trust_label
TESTED 2026-09-23 (one internal run) · VERIFIED 2026-10-01 (model names, tags, papers and terms re-checked at public sources) · UPDATED 2026-10-01
state_as_of
2026-09-23 — a dated observation of ONE run, NOT a benchmark
language
human view: en · bulgarian edition: /academy/moduli/KW_I28_Multimodel_Council_BG.html
previous
KW_I27_Product_Layer_EU_Boundary.html
next
KW_I29_Five_Untested_Trials.html · Five trials we did not run
PURPOSE

One task, ten models (six subscription web UIs, one free external endpoint, three local models), one human arbiter. Claude did the analysis; it also answered, so it is not rated. The run observed rule discipline under a Bulgarian-language constraint, not raw capability. This module keeps what survived: collaboration patterns with their failure modes, how to expose unanimous error, a role pipeline, script-level filters that need no model, and a qualitative reading of the run. There is no leaderboard; treat everything as a dated observation of one run.

KEY CONCEPTS
COMMANDS / PATHS
CHECKLIST
NEXT MODULE

KW_I29_Five_Untested_Trials.html · Five trials we did not run · offer: Quick experiment (kagami.bg/stalbata/)

SOURCES
TAGS
multi-modelconsensus-errorbulgarianlanguage-driftevaluationpipeline
TESTED · 23.09.2026 VERIFIED · 01.10.2026 UPDATED · 01.10.2026

10 models in Bulgarian: what survived

One task, ten models, one arbiter. An observation as of 23.09.2026 — a single run, not a benchmark. We did not measure who is smartest but who keeps the rules when the language is Bulgarian and the truth is not obvious. Here is what survived: the patterns with their failures, how to catch unanimous error, and the pipeline that remained.

⏱ ~22 min Intermediate Local AI multi-model · observation as of 23.09.2026
Local models (Ollama)🔒 local Subscription web assistants🌐 global model Free external tier🌐 global model

01What you will learn

02Before you start

🔄
UPDATED · 01.10.2026 — what changed
The lesson was reworked as an observation as of a date, not a benchmark. Removed: the 1-to-10 ranking with our own weights (no reproducible method), the comparison with an earlier capability ranking, word counts and thinking times of individual models, internal paths and machine names. Added: a table with the exact model names from public sources, checked on 01.10.2026; the paper citations (Du, Dhuliawala, Leviathan, Nature) were checked; OpenAI's terms were checked against the Europe Terms of Use, updated on 16.01.2026. The conclusions agree with the reference matrix in I26.

03Steps

  1. The run — what we ran and what it does NOT measure

    The task held five questions on multi-model systems: which collaboration patterns pay off, how to catch unanimous error, how to assign roles, how to work through subscriptions without an API, and when a council is not worth it. The rules were strict: 600 words max, banned buzzwords, "I don't know" instead of invention, say so when a pattern's name is your own, and one closing sentence on your own weakness.

    Model / serviceWherePublic fact as of 01.10.2026
    ChatGPT · Gemini · Grok · Perplexity · DeepSeek🌐 subscription web assistantWe do not know which model sits behind the interface (free tier, automatic routing).
    Vibe🌐 subscription web assistantMistral's assistant; its entry point remains chat.mistral.ai (vendor documentation).
    Nemotron 3 Ultra🌐 free external tierOpen weights from NVIDIA: 550B parameters in total, 55B active — per the model name on Hugging Face; licence "other"; the language list on the card does not include Bulgarian. The free tier of the external service changes — ⚠️ its availability today is not confirmed.
    gemma4:26b🔒 localMixture of experts (≈4B active parameters), 256K context, text and image — per the Ollama library.
    gemma3:27b🔒 local128K context, text and image; Gemma Terms of Use — per the Ollama library.
    gpt-oss:20b🔒 localOpenAI's open reasoning model, Apache 2.0 licence, 128K context — per the Ollama library.

    Claude did the analysis — it also answered, so it is not among the models rated. The arbiter was a human.

    📏
    The rules are the instrument
    A broken rule is data: an exceeded limit, an invented name passed off as established, a Russian word in Bulgarian text. That makes discipline measurable without needing a "right answer".
    ⚠️
    What this run does NOT measure
    One run. Unknown versions behind the web UIs. Unequal thinking time between models. We look at the answers, not at the models in general — the result today may differ. Who is right when the truth is known is a matter for separate trials — see I29.
  2. Patterns that survive

    For each pattern the failure matters more than the success. Two names are established in the literature; the rest are descriptive.

    PatternGood forWhere it fails
    Writer–critic (different models)document structure, code, argumentthe critic has nothing to compare against and invents problems; after round two the text gets worse. Fix: one round, a defect list, no rewrite
    Debate between models (Multiagent Debate, Du et al., 2023)logic and arithmetic with checkable stepsconformity: the lone correct voice yields to the most confident one; debate amplifies confidence, not truth
    Chain-of-Verification (Dhuliawala et al., 2023)facts, deadlines, articles of law, sumsan error in the framing (the wrong question); a verifier carrying the same wrong fact
    Artifact pipelinedecomposable work with clear intermediate productsan early wrong premise gets polished down the chain; the validator must be able to send back a specific stage
    🗳️
    Voting is not verification
    In an earlier trial of ours most of the models were wrong and the lone voice was right — for the wrong reason. A "keep only what 2/3 said" pattern (proposed by one participant) is a machine for amplifying exactly that error.
    🏷️
    An invented name = the costliest error
    Participants passed their own names off as established. "Speculative decoration" sounds like speculative decoding (Leviathan et al., 2022 — speeding up inference with a small model that proposes and a large one that verifies) but has nothing to do with it. "Entropy test" inverts the logic of semantic entropy (Nature, 2024), where disagreement between the meanings of several answers is the signal of uncertainty, not similarity. A plausible name makes someone build for a whole day.
    1️⃣
    When one model beats a council
    Text in one voice; code with a runnable test (the test is the arbiter); machine-checkable tasks; work on one long own document. Without an external validator five models add confidence, not knowledge.
  3. How to catch unanimous error

    The most dangerous case: everyone agrees and everyone is wrong, because they learned from similar texts. Agreement between such models is correlation, not evidence. Bulgarian specifics are thin in the training text, so agreement is often a foreign template — a Russian or generic EU rule, delivered with confidence. Measure independent grounds, not the vote count.

    MechanismHow it works
    Falsifiersees only the task and the majority thesis, not the reasoning; output: claim / condition where it is false / test; at most 5 defects
    Control questionsquestions with known answers from the same domain; a model that fails them is dropped from synthesis
    Paraphrasedifferent phrasings to different models; agreement after paraphrase weighs more
    Red linea tool (primary source, test, calculator) says not-X while the models say X → shown before synthesis
    Shared citationfour models citing the same (possibly invented) source are one error, not four votes
    Verifier strengththe verifier is never weaker than the generator — a weak one misses a strong one's subtle errors
    🎭
    Devil's advocate without a format = theatre
    "Assume everyone is wrong" without an output format produces eloquent disagreement. The measure is a concrete line to check, not eloquence.
    🔧
    Rule
    The voice that breaks agreement is not another model but a tool with no opinion: a primary source, a compiler, a control question, a human.
  4. The pipeline and the roles

    Four kinds of model: strong local (slow, cleaner Bulgarian), fast local (weak language), large external (strong, rate-limited), multimodal. The pipeline that remained:

    StepWhoJob
    1. Inputmultimodalonly for an image or PDF — turns it into text
    2. Routingfast localclassifies, extracts claims, runs control questions; writes no draft
    3. Independent roundstrong local + large externalblind: a Bulgarian draft / notes on the plan, not prose
    4. Filtersscript, no modellanguage, quarantine, control questions
    5. Falsificationlarge externalthesis only, up to 5 defects
    6. Verificationtoolsprimary source, test, calculator
    7. Synthesisstrong local, fresh contextonly on the output of filters and verification
    8. Arbiterhumanthe red lines and the "unverified" column
    ⚖️
    Open conflict
    The strong local model writes the draft and does the synthesis, yet the synthesizer should not be the author — it forgives itself. Two ways out: the large external model writes the draft (spends its limit on prose) or a fresh-context compromise. Settled by trial, not by opinion — see I29.
    🚫
    Dropped
    A weak model writing the draft (three local models proposed it unanimously — a learned template, refuted in practice: weak models rewrite the task). Voting by 2/3. Automating a subscription web chat.
    🔐
    Without an API — what is legitimate
    Manual copying with a single pack between chats; the vendor's API for repeatability. A bot, script or agent driving a subscription web UI, a session-cookie proxy, bypassing bot protection — all of it risks the terms of service. OpenAI's Europe Terms of Use (updated 16.01.2026) explicitly forbid automatically or programmatically extracting data and output; for other vendors, check their current terms.
  5. Filters with no model

    The cheapest and most reliable part of the pipeline is not a model but a script. Many participants listed "removing Russianisms" as a job for the strong model. None proposed how to catch it automatically — and in this run five of the ten made language errors in their own answer. Drift comes mostly in Bulgarian letters, so a check for ы/э/ё alone misses most of it.

    ModelWhat we sawWhich layer catches it
    Vibe"скорость" (Russian)3 (spelling)
    Nemotron 3 Ultra"Сilen", "структурa" (a Latin letter inside Cyrillic)2 (mixed script)
    gpt-oss:20b"черновка", "низка", "диссонанс" (Russian forms)3 (spelling)
    gemma4:26b"къде" instead of "където" (a calque)none — a calque, needs a human
    gemma3:27b"приложете" instead of "приложат" (grammar)none — grammar
    python · a three-layer language filter
    # Layer 1: letters absent from Bulgarian
    import re, subprocess
    RU = re.compile(r"[ыэёЫЭЁ]")
    # Layer 2: a word with Cyrillic AND Latin letters
    def mixed(words):
        return [w for w in words if re.search(r"[А-Яа-я]", w) and re.search(r"[A-Za-z]", w)]
    # Layer 3: spelling against the bg_BG dictionary minus protected terms
    def unknown(text, protected):
        out = subprocess.run(["hunspell", "-d", "bg_BG", "-l"], input=text,
                             capture_output=True, text=True).stdout.split()
        return [w for w in out if re.search(r"[А-Яа-я]", w) and w not in protected]
    🧯
    Protected-terms list
    Aggressive cleaning deletes technical terms and shifts meaning (a failure named by gemma4:26b). Before layer 3 — a list of words the filter never touches.
    🧪
    Quarantine for unsourced numbers and names
    Every number, threshold, latency or technique name that is not in the task and comes without a source goes automatically into an "unverified" column. It would have caught "0.4 cosine = risk", "≈10 ms" and "20 requests/min" — all invented by participants. Output is also cleaned of UI leftovers ("Ask", "Explain").
  6. What we observed — without a ranking

    The order is alphabetical, not by quality. We describe what happened in this run and what of it entered the system. We give no scores: we have no reproducible method, and a model was often good at one thing and weak at another.

    ModelWhat we noticedWhat enters the system
    ChatGPT 🌐no invented names and no unsafe adviceconfidence per claim
    DeepSeek 🌐no invented names and no unsafe advice; says "I don't know"blind judge
    Gemini 🌐an invented technique ("card blocking"); advice to use a session-cookie proxy — risky under the terms of service—
    gemma3:27b 🔒invented nothing, but added nothing new either; Selenium listed as "possible"—
    gemma4:26b 🔒an "entropy test" with inverted logic; a calque in the languagethe language-filter failure
    gpt-oss:20b 🔒invented latencies and a "dissonance detection"; scraping as a method; drift into Russian—
    Grok 🌐no inventions and no unsafe advicered line "the tool says not-X"; a shared citation = one error
    Nemotron 3 Ultra 🌐"0.4 cosine" and "speculative decoration" passed off as fact; an idea for bypassing bot protection; mixed scripta validator that writes breaking tests; the ast-grep tool
    Perplexity 🌐exceeded the 600-word limit; passed off names without a caveata falsifier that sees only the thesis; a format; up to 5 defects
    Vibe 🌐a false "the name is mine"; "grey" user scripts; language driftcontrol questions; paraphrase

    Conclusions

    Discipline ≠ capability. A model that is strong at general tasks is not necessarily the one that keeps the rules. The two axes measure different things — a signal, not a verdict.

    EU fitness ≠ clean Bulgarian. A European model drifted into Russian. Language is a separate criterion.

    A voice with a primary source beats a voice with an opinion. The model that searches before answering quoted the terms of service accurately.

    A live example of unanimous error. All three local models put the weak model on the first draft. That is the "draft → polish" template learned from similar texts. It is visible only because the arbiter has experience outside the corpus.

    🖥️
    The local stack after the run
    gpt-oss:20b — out of Bulgarian tasks. gemma4:26b — a candidate for a fast classifier (not a verdict yet). gemma3:27b — added nothing new to the council. No local model proved fit to be a critic or judge. Verdicts per model are in the I26 reference matrix.
    📅
    Best before: 23.09.2026
    Models change within weeks. This is our experience from a single run, not an independent assessment. Before a decision involving personal data or a quote, repeat a small trial with your own documents and write down the date.

04Check

1. Four models agree on a fact from a Bulgarian regulation. What do you do?

2. Who writes the first draft in the pipeline?

3. What does the falsifier see?

4. Which Russian word is NOT caught by the ы/э/ё check?

05What's next

06Sources

  1. Ollama library — names, tags and licences of the local models: gemma4, gemma3, gpt-oss.
  2. NVIDIA Nemotron 3 Ultra — the model card on Hugging Face (licence, language list).
  3. Mistral Vibe — the vendor's documentation.
  4. OpenAI — Europe Terms of Use (updated 16.01.2026) — the ban on automatically or programmatically extracting data and output.
  5. Du et al., 2023 — Multiagent Debate · Dhuliawala et al., 2023 — Chain-of-Verification · Leviathan et al. — Speculative Decoding · Nature, 2024 — semantic entropy.
  6. Hunspell and Debian: hunspell-bg — the spelling check for layer 3; ast-grep — the structural code-search tool.
  7. Companion: I26 · Which model for what.
  8. KAGAMI's own observation, 23.09.2026: one task, ten models. Our experience from a single run, not a benchmark.