10 models in Bulgarian: what survived
One task, ten models, one arbiter. An observation as of 23.09.2026 — a single run, not a benchmark. We did not measure who is smartest but who keeps the rules when the language is Bulgarian and the truth is not obvious. Here is what survived: the patterns with their failures, how to catch unanimous error, and the pipeline that remained.
01What you will learn
- Which collaboration patterns between models pay off — and where each one fails.
- How to catch an error that every model agrees on, and why voting is not verification.
- How to assign roles in a pipeline of models, and when one model beats a council.
- Which filters catch drift into Russian and invented numbers without a model — with a script.
02Before you start
- Read this in the right frame: this is an observation, not a ranking. One run on one day (23.09.2026), with no reproducible method for scoring. That is why there are no points and no 1-to-10 order here — only what we noticed and what entered the system.
- You do not need all the models. The idea works with two or three — what matters is that they come from different families.
- For the filter in step 5: Python 3 and
hunspellwith a Bulgarian dictionary (thehunspell-bgpackage on Debian) 🔒 local. - You know what a "council" means here: the same task given to several models, and a human who compares the answers.
03Steps
-
The run — what we ran and what it does NOT measure
The task held five questions on multi-model systems: which collaboration patterns pay off, how to catch unanimous error, how to assign roles, how to work through subscriptions without an API, and when a council is not worth it. The rules were strict: 600 words max, banned buzzwords, "I don't know" instead of invention, say so when a pattern's name is your own, and one closing sentence on your own weakness.
Model / service Where Public fact as of 01.10.2026 ChatGPT · Gemini · Grok · Perplexity · DeepSeek 🌐 subscription web assistant We do not know which model sits behind the interface (free tier, automatic routing). Vibe 🌐 subscription web assistant Mistral's assistant; its entry point remains chat.mistral.ai (vendor documentation). Nemotron 3 Ultra🌐 free external tier Open weights from NVIDIA: 550B parameters in total, 55B active — per the model name on Hugging Face; licence "other"; the language list on the card does not include Bulgarian. The free tier of the external service changes — ⚠️ its availability today is not confirmed. gemma4:26b🔒 local Mixture of experts (≈4B active parameters), 256K context, text and image — per the Ollama library. gemma3:27b🔒 local 128K context, text and image; Gemma Terms of Use — per the Ollama library. gpt-oss:20b🔒 local OpenAI's open reasoning model, Apache 2.0 licence, 128K context — per the Ollama library. Claude did the analysis — it also answered, so it is not among the models rated. The arbiter was a human.
📏The rules are the instrumentA broken rule is data: an exceeded limit, an invented name passed off as established, a Russian word in Bulgarian text. That makes discipline measurable without needing a "right answer".⚠️What this run does NOT measureOne run. Unknown versions behind the web UIs. Unequal thinking time between models. We look at the answers, not at the models in general — the result today may differ. Who is right when the truth is known is a matter for separate trials — see I29. -
Patterns that survive
For each pattern the failure matters more than the success. Two names are established in the literature; the rest are descriptive.
Pattern Good for Where it fails Writer–critic (different models) document structure, code, argument the critic has nothing to compare against and invents problems; after round two the text gets worse. Fix: one round, a defect list, no rewrite Debate between models (Multiagent Debate, Du et al., 2023) logic and arithmetic with checkable steps conformity: the lone correct voice yields to the most confident one; debate amplifies confidence, not truth Chain-of-Verification (Dhuliawala et al., 2023) facts, deadlines, articles of law, sums an error in the framing (the wrong question); a verifier carrying the same wrong fact Artifact pipeline decomposable work with clear intermediate products an early wrong premise gets polished down the chain; the validator must be able to send back a specific stage 🗳️Voting is not verificationIn an earlier trial of ours most of the models were wrong and the lone voice was right — for the wrong reason. A "keep only what 2/3 said" pattern (proposed by one participant) is a machine for amplifying exactly that error.🏷️An invented name = the costliest errorParticipants passed their own names off as established. "Speculative decoration" sounds like speculative decoding (Leviathan et al., 2022 — speeding up inference with a small model that proposes and a large one that verifies) but has nothing to do with it. "Entropy test" inverts the logic of semantic entropy (Nature, 2024), where disagreement between the meanings of several answers is the signal of uncertainty, not similarity. A plausible name makes someone build for a whole day.1️⃣When one model beats a councilText in one voice; code with a runnable test (the test is the arbiter); machine-checkable tasks; work on one long own document. Without an external validator five models add confidence, not knowledge. -
How to catch unanimous error
The most dangerous case: everyone agrees and everyone is wrong, because they learned from similar texts. Agreement between such models is correlation, not evidence. Bulgarian specifics are thin in the training text, so agreement is often a foreign template — a Russian or generic EU rule, delivered with confidence. Measure independent grounds, not the vote count.
Mechanism How it works Falsifier sees only the task and the majority thesis, not the reasoning; output: claim / condition where it is false / test; at most 5 defects Control questions questions with known answers from the same domain; a model that fails them is dropped from synthesis Paraphrase different phrasings to different models; agreement after paraphrase weighs more Red line a tool (primary source, test, calculator) says not-X while the models say X → shown before synthesis Shared citation four models citing the same (possibly invented) source are one error, not four votes Verifier strength the verifier is never weaker than the generator — a weak one misses a strong one's subtle errors 🎭Devil's advocate without a format = theatre"Assume everyone is wrong" without an output format produces eloquent disagreement. The measure is a concrete line to check, not eloquence.🔧RuleThe voice that breaks agreement is not another model but a tool with no opinion: a primary source, a compiler, a control question, a human. -
The pipeline and the roles
Four kinds of model: strong local (slow, cleaner Bulgarian), fast local (weak language), large external (strong, rate-limited), multimodal. The pipeline that remained:
Step Who Job 1. Input multimodal only for an image or PDF — turns it into text 2. Routing fast local classifies, extracts claims, runs control questions; writes no draft 3. Independent round strong local + large external blind: a Bulgarian draft / notes on the plan, not prose 4. Filters script, no model language, quarantine, control questions 5. Falsification large external thesis only, up to 5 defects 6. Verification tools primary source, test, calculator 7. Synthesis strong local, fresh context only on the output of filters and verification 8. Arbiter human the red lines and the "unverified" column ⚖️Open conflictThe strong local model writes the draft and does the synthesis, yet the synthesizer should not be the author — it forgives itself. Two ways out: the large external model writes the draft (spends its limit on prose) or a fresh-context compromise. Settled by trial, not by opinion — see I29.🚫DroppedA weak model writing the draft (three local models proposed it unanimously — a learned template, refuted in practice: weak models rewrite the task). Voting by 2/3. Automating a subscription web chat.🔐Without an API — what is legitimateManual copying with a single pack between chats; the vendor's API for repeatability. A bot, script or agent driving a subscription web UI, a session-cookie proxy, bypassing bot protection — all of it risks the terms of service. OpenAI's Europe Terms of Use (updated 16.01.2026) explicitly forbid automatically or programmatically extracting data and output; for other vendors, check their current terms. -
Filters with no model
The cheapest and most reliable part of the pipeline is not a model but a script. Many participants listed "removing Russianisms" as a job for the strong model. None proposed how to catch it automatically — and in this run five of the ten made language errors in their own answer. Drift comes mostly in Bulgarian letters, so a check for ы/э/ё alone misses most of it.
Model What we saw Which layer catches it Vibe "скорость" (Russian) 3 (spelling) Nemotron 3 Ultra "Сilen", "структурa" (a Latin letter inside Cyrillic) 2 (mixed script) gpt-oss:20b "черновка", "низка", "диссонанс" (Russian forms) 3 (spelling) gemma4:26b "къде" instead of "където" (a calque) none — a calque, needs a human gemma3:27b "приложете" instead of "приложат" (grammar) none — grammar python · a three-layer language filter# Layer 1: letters absent from Bulgarian import re, subprocess RU = re.compile(r"[ыэёЫЭЁ]") # Layer 2: a word with Cyrillic AND Latin letters def mixed(words): return [w for w in words if re.search(r"[А-Яа-я]", w) and re.search(r"[A-Za-z]", w)] # Layer 3: spelling against the bg_BG dictionary minus protected terms def unknown(text, protected): out = subprocess.run(["hunspell", "-d", "bg_BG", "-l"], input=text, capture_output=True, text=True).stdout.split() return [w for w in out if re.search(r"[А-Яа-я]", w) and w not in protected]🧯Protected-terms listAggressive cleaning deletes technical terms and shifts meaning (a failure named by gemma4:26b). Before layer 3 — a list of words the filter never touches.🧪Quarantine for unsourced numbers and namesEvery number, threshold, latency or technique name that is not in the task and comes without a source goes automatically into an "unverified" column. It would have caught "0.4 cosine = risk", "≈10 ms" and "20 requests/min" — all invented by participants. Output is also cleaned of UI leftovers ("Ask", "Explain"). -
What we observed — without a ranking
The order is alphabetical, not by quality. We describe what happened in this run and what of it entered the system. We give no scores: we have no reproducible method, and a model was often good at one thing and weak at another.
Model What we noticed What enters the system ChatGPT 🌐 no invented names and no unsafe advice confidence per claim DeepSeek 🌐 no invented names and no unsafe advice; says "I don't know" blind judge Gemini 🌐 an invented technique ("card blocking"); advice to use a session-cookie proxy — risky under the terms of service — gemma3:27b 🔒 invented nothing, but added nothing new either; Selenium listed as "possible" — gemma4:26b 🔒 an "entropy test" with inverted logic; a calque in the language the language-filter failure gpt-oss:20b 🔒 invented latencies and a "dissonance detection"; scraping as a method; drift into Russian — Grok 🌐 no inventions and no unsafe advice red line "the tool says not-X"; a shared citation = one error Nemotron 3 Ultra 🌐 "0.4 cosine" and "speculative decoration" passed off as fact; an idea for bypassing bot protection; mixed script a validator that writes breaking tests; the ast-grep tool Perplexity 🌐 exceeded the 600-word limit; passed off names without a caveat a falsifier that sees only the thesis; a format; up to 5 defects Vibe 🌐 a false "the name is mine"; "grey" user scripts; language drift control questions; paraphrase Conclusions
Discipline ≠ capability. A model that is strong at general tasks is not necessarily the one that keeps the rules. The two axes measure different things — a signal, not a verdict.
EU fitness ≠ clean Bulgarian. A European model drifted into Russian. Language is a separate criterion.
A voice with a primary source beats a voice with an opinion. The model that searches before answering quoted the terms of service accurately.
A live example of unanimous error. All three local models put the weak model on the first draft. That is the "draft → polish" template learned from similar texts. It is visible only because the arbiter has experience outside the corpus.
🖥️The local stack after the rungpt-oss:20b — out of Bulgarian tasks. gemma4:26b — a candidate for a fast classifier (not a verdict yet). gemma3:27b — added nothing new to the council. No local model proved fit to be a critic or judge. Verdicts per model are in the I26 reference matrix.📅Best before: 23.09.2026Models change within weeks. This is our experience from a single run, not an independent assessment. Before a decision involving personal data or a quote, repeat a small trial with your own documents and write down the date.
04Check
1. Four models agree on a fact from a Bulgarian regulation. What do you do?
2. Who writes the first draft in the pipeline?
3. What does the falsifier see?
4. Which Russian word is NOT caught by the ы/э/ё check?
05What's next
06Sources
- Ollama library — names, tags and licences of the local models: gemma4, gemma3, gpt-oss.
- NVIDIA Nemotron 3 Ultra — the model card on Hugging Face (licence, language list).
- Mistral Vibe — the vendor's documentation.
- OpenAI — Europe Terms of Use (updated 16.01.2026) — the ban on automatically or programmatically extracting data and output.
- Du et al., 2023 — Multiagent Debate · Dhuliawala et al., 2023 — Chain-of-Verification · Leviathan et al. — Speculative Decoding · Nature, 2024 — semantic entropy.
- Hunspell and Debian: hunspell-bg — the spelling check for layer 3; ast-grep — the structural code-search tool.
- Companion: I26 · Which model for what.
- KAGAMI's own observation, 23.09.2026: one task, ten models. Our experience from a single run, not a benchmark.