克里米亞韃靼語建庫的資料教訓：語言過濾器難辨近親語言

In the first post of this series I wrote that Crimean Tatar had no dataset good enough for serious work, and that I started by fixing an existing speech set. In the second a recogniser learned to hear our speech, and in the third a synthetic voice read a whole novel. The question I got most after those: if there was no data, what did any of it train on?

This post is the answer, organised as the lessons I'd hand to anyone building a corpus for a language the internet barely knows. Most of them are not about models. They're about data that looks fine and isn't.

Same caveat as before, a bit stricter: I don't name sources, archives, outlets, channels or the people who shared material with us, or our arrangements with them. No model names, filter thresholds or internal scripts either. Volumes are given as orders of magnitude. Metrics, methods and mistakes are all in.

What "data" meant here

We had printed books and periodicals from many decades, broadcast recordings (news, programmes, studio talk, interviews), professionally narrated audiobooks, dictionaries, my own recordings, and what teachers bring us: dialogues, corrections, voice recordings.

Almost none of that was a dataset. Two products were needed. For speech models: audio paired with the exact text spoken in it. For the language model: clean text you can trust. A book on a shelf is neither. A broadcast is hours of audio with no transcript. An audiobook has audio and text living separately, if the text exists digitally at all.

Each kind of material came with its own failure mode. Here they are in the order I hit them.

Lesson 1: inventory before you build the expensive pipeline

The plan for books looked obvious. Most older Crimean Tatar books exist only as scans, so: OCR. On our language OCR is slow and error-prone, especially on letters Russian doesn't have, so this was going to be the expensive branch.

Before running anything I counted what was actually in the archive, by page:

Among Crimean Tatar pages alone, 79% already carried a text layer: real text under the page image, extractable for free. OCR was only needed for the remaining fifth. The expensive branch was the smaller half of the prize.

Count first. The question "how do we OCR all of this?" was aimed slightly wrong.

Lesson 2: free text is broken in boring, mechanical ways

"There's text" doesn't mean "the text is good." Three defect classes showed up immediately, and none of them came from our recognition. The layers had been produced long ago by someone else's software, often old OCR, and got damaged then or in later file conversions.

Soft sign instead of hard sign. къаплангьан instead of къаплангъан (Latin: qaplanğan), догьрумы instead of догърумы (doğrumı). This is what a Cyrillic engine trained on Russian does to our language. The good news: кь and гь never occur in Crimean Tatar Cyrillic, so the repair is a mechanical substitution. I left нь alone, because it legitimately appears in Russian loanwords.

Wrong code page. Latin pages read as tatarlýq, þöyle, qorquya tüþüp. Turkish-encoded bytes had been decoded as Western European, so ı became ý, ş became þ, ğ became ð. Reverse the mapping and you get tatarlıq, şöyle, qorquya tüşüp.

Duplicates. The same works in different editions, anthologies, and file names. The second post had the audio version of this: four audiobooks stored twice under unrelated names. Text did it too. File names caught almost nothing. Comparing content, each page against the already accepted ones by eight-word shingles, found 7.0% repeats.

The one habit from this section I'd push on anyone: run every mechanical repair on known-good text first, and require it to change nothing. My soft-sign fix went over a clean set before it touched the archive. The page metric before and after was identical to four decimal places, zero characters changed. Only then did it run across the archive, where it fixed tens of thousands of pages. A repair that touches clean text is a bug, and that's the cheapest place to catch it.

Lesson 3: score pages, not books, and calibrate the score

The unit of decision is the page. In one volume the Cyrillic part was clean and the Latin part was an old failed OCR whose layer was pure noise. A per-book verdict forces you to either drop good pages or accept bad ones.

To score a page nobody has proofread, I used our dictionary: the share of tokens that start with a dictionary word. Prefix, not exact match. Crimean Tatar is agglutinative, ending after ending stacks on a root, and exact match could not tell a good book (3% hits) from a broken one (2%). With the prefix rule it becomes a real instrument. Two more signals sit next to it: the share of garbage characters, and the share of кь/гь/нь among all of къ/гъ/нъ/кь/гь/нь, computed only on pages with enough of those digraphs, because on a sparse page the ratio is noise. The signals are reported separately, never averaged.

Thresholds were calibrated, not picked: on known-clean digital text, on known-good OCR with a measured CER around 0.3%, on an illustrated dictionary (legitimately sparse text), and on a known-broken book where a Cyrillic model had read Latin script. Then negative controls. Six random Russian books: 100% of pages rejected. Six Turkish books: 74% rejected, but two of the six passed.

A lexicon of your language will partly accept a close relative. That's the honest limit of the method, and it's the setup for the next lesson.

The payoff: trainable text grew roughly tenfold over the corpus core before this step, and most of that growth needed no OCR. The expensive part turned out to be deciding whether to trust text that already existed.

Lesson 4: language ID can't see a sister language

One day our language model answered me with "Ayt, men dinlemeye hazırım." In Crimean Tatar it should be "Ayt, men diñlemege azırım." Same roots, Turkish shape.

I turned the complaint into a measurement. On the same 524 evaluation prompts, the share of Latin tokens with a Turkish shape was 2.02% for the new version against 0.27% for the previous one, 7.5 times more. The instrument is anchored at both ends: known-clean Crimean Tatar prose scores 0.03%, real Turkish prose scores 11.75%. On the largest free-translation set, almost one in ten dative verbal nouns came out with Turkish -meye instead of our -mege.

The cause was in the data. Some periodicals published outside Crimea carry real Crimean Tatar pages and, in the same issues, whole pages in Turkish. Those pages had flowed into the corpus.

Why didn't the language filter stop them? A language-ID filter answers "is this our language or not?" from function words, endings and character statistics. For Turkish the answer is nearly always "probably yes": too many shared roots and function words. My paragraph-level filter, run over the whole corpus, flagged more than ten thousand pieces for removal: Russian, Ukrainian, English. Turkish features showed up in 40 of them. That's not a tuning problem. "Ours versus everything else" cannot separate two languages that share most of their vocabulary.

What worked was the inverse: detect forms that never occur in your language.

A toy version of the idea:

import re

# forms that are Turkish and never Crimean Tatar (illustrative, not the real list)
TURKISH_SHAPES = [
    re.compile(r"\w+yor(um|sun|uz|lar)?\b"),   # present tense -yor
    re.compile(r"\bdeğil\b"),                  # ours is "degil"
    re.compile(r"\biki\b"),                    # ours is "eki"
    re.compile(r"\w+[mn][ea]y[ea]\b"),         # dative -meye/-maya; ours is -mege/-mağa
]

def turkish_share(tokens):
    hits = sum(any(p.fullmatch(t) for p in TURKISH_SHAPES) for t in tokens)
    return hits / max(len(tokens), 1)

The real detectors are more careful than this, and every one was validated in both directions: silent on clean Crimean Tatar, firing on real Turkish. A second detector, for other Turkic languages, came later and found a few dozen pieces written entirely in Kazakh, Tatar or Uzbek. Neither language ID nor the Turkish detector had seen them.

Several thousand fragments went to quarantine. Nothing was deleted. Each is kept as it was, with its origin and the rule that fired, so any call can be reversed.

If you work on a language with a big, well-resourced sibling, assume the sibling is in your corpus, and build detectors from the sibling's side.

Lesson 5: a normaliser that makes plausible mistakes is worse than none

The tempting next step: don't drop Turkish spellings, normalise them. k → q, n → ñ, and so on. I tested it and shelved it.

The program fixed shapes without understanding morphology. balaların ("of the children", genitive) became balalarıñ, which means "your children." The correct form is balalarnıñ. Every such correction produced a real Crimean Tatar word with a different meaning. The dictionary check passes it. The Turkish detector passes it. Only a reader catches it.

Errors that look like valid data are the most expensive kind, because nothing downstream will flag them.

Lesson 6: decide what you're filtering for before you filter dialects out

The flip side. Our dictionaries mark some words as dialect, for example the south-coast ben instead of literary men ("I"). The Turkish detectors flag them, naturally. I went through the full list and decided these stay: they're varieties of our own language, even where they coincide with Turkish.

The distinction moved to evaluation. Dialect forms remain in the training corpus; in the model's answers they're counted as their own category. I expect the model to answer in the literary language, but it can learn from everything the language actually lives in.

Lesson 7: two models that disagree differently can label audio, but agreement isn't proof

There was an audiobook several hours long: good narrator, clean recording, and no digital text of that edition anywhere. Forced alignment against a known text, the trick the project started from (first post), wasn't possible.

So: our recogniser transcribes it, then a second model of a completely different architecture, trained separately, transcribes the same audio. A segment enters the corpus only where both transcripts nearly coincide. Models that fail differently rarely fail the same way. After the main recogniser improved and segmentation got more precise, the same book yielded 2.6 times more usable audio, and the segments where the old model had looped, repeating one phrase fifty times over, were gone.

The caveat I keep in front of me: two wrong models can agree, and there's no book to check against. Those segments can only be listened to, which is why every doubtful piece now goes to a human ear.

Lesson 8: a confidence filter keeps what the model already knows

Next came broadcasts, over a thousand hours. Same pattern: transcribe, keep only what the model is confident about and the second model agrees with. The strict filter kept about a quarter.

Then I looked at what passed, as shares of speech hours:

News passed three times as readily as conversation. Newsreaders read from a script, clearly, and the recogniser already handles them well. Conversation, with interruptions, false starts and real-life pace, got through only in small part. A confidence filter systematically keeps what the model already does well and discards exactly what it needs to learn. (Interviews look even worse in the raw numbers, but mostly for a different reason: I'd removed most of them by rule before the filter. Oral histories are held out as an exam, and interviews mixed with other languages are excluded.)

The fix for the next round: cap the share of news, and let conversation through on a softer threshold.

Whenever you filter pseudo-labels by confidence, break the survivors down by class. If the mix shifted towards the easy class, your filter is a selection bias with a threshold.

Lesson 9: if a model wrote your references, they're a hypothesis

The test transcripts for real speech had been produced by the machine, an earlier version of the recogniser. Any new model that disagreed with it got penalised automatically, even when it was right.

I took the segments where the models disagreed and listened to them myself, blind, not knowing which transcript was whose, writing down what I actually heard. It took many hours. The machine references were wrong in about 15% of their words, and the new model was right noticeably more often than the old one.

Where things stand now: the recogniser on our server was trained on real speech, and our partners use it a lot. On 40 short real-microphone clips, scored against my own transcripts, WER fell from 18.8% to 12.1%, about a third. On long conversations (80 units from three recordings) old and new are level, 30.5% and 30.7%. The next round, with several times more conversational hours, came out level with production on the automatic sets, and my blind long-form comparison ended 25 to 24, no winner. "Add more hours" stopped paying off. The bottleneck is now the quality of labels on conversational speech, not the volume.

Lesson 10: synthetic speech can win the benchmark and lose the real test

Synthetic audio comes with a perfect transcript, because you chose the text before the model read it. I mixed synthetic speech in several voices into ASR training. Every synthetic clip was first transcribed by the recogniser, and the ones that didn't match their text were dropped by a threshold fixed in advance. About 5% fell out.

I wrote my prediction down before the run: a little synthetic speech might help, half would hurt. Wrong. On an exam of two audiobooks the model had never heard, one run per arm:

The gain held on all six degraded copies of the exam: noise, reverb, a telephone band, faster and slower speech. A general gain, though, not special noise robustness: overlapping voices are still the main failure.

The other half: that exam is audiobooks, clean reading in a quiet room. On recordings closer to real life, the synthetic-mix model looked worse than the real-only one. That check isn't final, because its references were also machine-made (see lesson 9) and each arm was trained once. But the synthetic-mix model didn't ship. Production runs the one trained on real speech, and on real speech it's better.

Lesson 11: automatic checks can't certify generated text

For text my rule is: it doesn't matter who wrote it, a person or a machine. What matters is that it's clean, correct Crimean Tatar. Every generated batch goes through the dictionary check, the Turkish detectors, the transliteration check, and spot reading.

I asked several large language models to write Crimean Tatar dialogues and read 26 of them blind. Zero were usable as written. The best could be rescued with edits; the rest were mostly Turkish with Uzbek and Kazakh mixed in. Worse, the automatic metrics barely separated the models. One I rejected entirely scored like the others, because my detectors at the time saw Turkish but not Kazakh or Uzbek. That's exactly why the other-Turkic detector from lesson 4 exists.

The second attempt was stricter: longer dialogues, a detailed brief, and every one passed every automatic check with zero flags. I read six blind. None was even worth correcting. I rewrote two myself; four were hopeless, "lots of words nobody would understand, and the wrong forms."

A clean score from your detectors tells you the text avoids the errors you already know about. It says nothing about the ones you haven't written a detector for yet.

I'll admit I was upset at that point. Synthetic data hadn't solved the problem, and we still don't have enough teachers. For both audio and text it's a useful supplement under supervision, not a substitute for people.

Where the people come in

Finding people turned out to be one of the hardest parts. Many agreed to help; only a few have the time to actually do it, and that has slowed the work a lot. We've posted calls on social media, and people who care have answered. We keep looking.

The rule for voices: the voices and recordings in our examples and in our synthetic voices are used with their owners' consent. If anyone changes their mind, their voice comes out, no argument.

Teachers work with us through a dedicated portal. They write dialogues for the model to learn conversation from, check corpus lines and mark what isn't Crimean Tatar, sort dictionary notes into Turkish and ours, correct machine drafts, and record their voices. Those recordings never entered training. A teacher on an ordinary microphone is the closest thing we have to real users, so it's the exam, and you don't train on the exam.

Partners who record oral histories sent us an interview with subtitles they checked by hand: conversational speech, local dialect, pauses, and a transcript written by a person. It's one of our hardest exams, and the model doesn't train on it either.

And my own voice and ears. The machine read the second book in our library in my voice (third post), and the doubtful segments that no automatic check can settle, I listen to myself.

The whole list, in one place

The texts were where nobody looked: under the scans, needing repair and verification rather than recognition. The audio was in broadcasts and audiobooks, but its text had to be pulled out by two models checking each other, with an eye on what the filter quietly preferred. Synthetic data added a little, and only under supervision.

The decisions that mattered most were made by a native speaker's ear: telling Turkish from dialect, hearing that the references were wrong in one word in six or seven, and saying "not our language" about dialogues every instrument had passed. No filter did any of that.

Fourth post in the series. Audio examples for this one: https://ai.ana-yurt.dev/en/blog/2026-10-otkuda-dannye/.

Want to help? What the project needs most is native speakers. Listen to a short clip and say whether the machine transcribed it right. Read a page and mark which word is Turkish and which is ours, from a dialect. Record a few minutes of reading in your own voice. No programming, just knowing your language well; in this post the most important calls were made by ear. Write to ai@ay.mw or on Telegram: t.me/ana_yurt_dev.

If you'd rather support the work financially, you can do it here: ko-fi.com/anayurt. That support turns into new recordings of living speech, and those need to be made now, while there are still people to learn from.