企業選擇阿拉伯語大語言模型，應看自建評測而非排行榜

Four serious Arabic models now come out of the Gulf: Falcon-H1-Arabic from Abu Dhabi, ALLaM from Saudi Arabia, Fanar 2 from Qatar and Jais 2 from the UAE. None of them can be picked from a leaderboard, because the Open Arabic LLM Leaderboard measures academic Arabic, not your tickets, your policy documents or your customers' dialect. The two things that do transfer to your organisation are both measurable in about a week: a 200 to 500 item eval set built from your own material, and the tokenization economics that decide what each Arabic conversation really costs in money, latency and GPU memory.

If you are building Arabic-language customer service, a government services assistant or an internal knowledge bot in the GCC, you face a decision that looks like a model choice but is actually two measurements. First: does a regional open-weight model score within your tolerance on your content, with your product names, place names, dialects and policy language? Second: what does an Arabic token actually cost you, in API spend, time to first token, KV-cache footprint and concurrent sessions per GPU? Answer both and the decision makes itself. Skip them and you will either overpay a frontier API for years or ship a model that fails quietly on Gulf dialect.

The field in late 2026

Four model families are credible candidates for enterprise Arabic work today. All of them are open weight unless noted, which means you can run them on your own GPUs and keep data in-country.

Primary sources: the Falcon-H1-Arabic announcement and release blog, the Fanar-2-27B-Instruct model card, the ALLaM-7B-Instruct-preview model card, the HUMAIN Chat launch and the Jais 2 documentation.

The fourth candidate in every real procurement is a frontier API model prompted in Arabic. It belongs in the comparison, because on raw Arabic quality it often wins. The question is by how much, and what that gap costs.

Why the leaderboard cannot pick your model

The Open Arabic LLM Leaderboard is genuinely good work. It runs reproducible evaluations through the EleutherAI lm-evaluation-harness Arabic task set, so a score is a score, not a vendor claim. TII even documents that its own vLLM-based runs land within about one point of the leaderboard's numbers.

But read what it measures: knowledge, reading comprehension and reasoning over web-derived Arabic text, mostly Modern Standard Arabic. Your workload is not that. It is a customer writing "ما أقدر أرسّع الفاتورة من التطبيق" (I can't pay the bill faster from the app, in Gulf dialect), a code-switched complaint ("الـ bundle خلص، كيف أعمل top-up؟"), and questions about a product name that appears nowhere in any pretraining corpus. Leaderboard rank measures none of the things that will make your assistant succeed or embarrass you.

This is not an argument against the leaderboard. It is the screening stage: it tells you which models are worth a week of your team's time. The final stage happens on your data.

What transfers: build a 300-item eval set in three days

The single highest-leverage artefact in an Arabic LLM project is a small, versioned eval set built from your own material. Two hundred items is a floor, five hundred is plenty; beyond that the marginal information per item collapses.

Where the items come from:

1. Real tickets and chat logs (redacted). Take them verbatim, dialect and typos included. Never rewrite them into MSA, because your users will not.
2. Policy and product documents. Turn paragraphs into questions: "What is the fee for an early exit from this plan?"
3. Named entities. Product names, service tiers, districts, mall names, government portals. Models hallucinate most confidently where pretraining data is thinnest, and nothing is thinner than your internal naming scheme.
4. Deliberately hard slices. Aim for roughly 40% formal MSA, 30% Gulf dialect, 30% code-switched Arabic/English, in the proportions your traffic actually shows.

A concrete mix for a 300-item set, drawn from a telecom support case:

• 90 items from the last two years of Arabic chat logs, sampled by intent cluster (billing, top-up, coverage, cancellation), including Emirati and Saudi dialect items
• 60 questions whose answers live only in your policy PDFs
• 60 entity-grounded questions where the correct answer must quote your exact product or tariff name
• 45 dialect and code-switched items, verbatim from logs, never normalised
• 45 adversarial items: questions about services you do not offer, questions that mix a real product name with a wrong assumption, and date and amount arithmetic from a bill

That is one analyst-day to sample, one subject-matter-expert day to write gold answers, one more to review. Gold answers should be short and checkable, not essays.

The scoring sheet and the judge

Score each response on five criteria, 0 to 2 each, so the maximum is 10 per item and a 300-item set yields a 3,000-point scale:

Two ways to score: a human reviewer (expensive, about 1 to 2 minutes per item, so a full pass on 300 items is a day) or a stronger model as judge. If you use a model as judge, check agreement first: have the human score a 50-item overlap sample and compute percent agreement on the Correctness criterion. Below roughly 85% agreement, your judge is not measuring your rubric, and you must fix the prompt before trusting it at scale.

Here is a copy-paste judge prompt that has the properties that matter: it forces evidence, it forbids generosity, and it outputs a parseable score.

You are evaluating a customer-service assistant's reply in Arabic.

QUESTION (verbatim from a real user, may contain Gulf dialect or
Arabic/English code-switching):
{question}

GOLD ANSWER (written by a domain expert):
{gold_answer}

ASSISTANT'S REPLY:
{reply}

Score each criterion 0, 1 or 2. Be strict: a score of 2 must be
clearly earned, not merely plausible.

1. correctness: 2 = matches the gold answer's facts, entities and
   amounts; 1 = partially right; 0 = wrong.
2. grounding: 2 = every claim is traceable to the gold answer or the
   supplied policy text; 1 = vague; 0 = invents facts.
3. dialect: 2 = correctly understands the question's register and
   replies naturally in it; 1 = understands but replies in stiff MSA;
   0 = misunderstands the dialect or code-switching.
4. language: 2 = natural, correct Arabic; 1 = awkward but usable;
   0 = broken Arabic or wrong language.
5. completeness: 2 = covers steps and caveats; 1 = usable with gaps;
   0 = misses something essential.

Reply ONLY with JSON:
{"correctness": n, "grounding": n, "dialect": n, "language": n,
 "completeness": n, "total": n,
 "fatal_error": "<one line, or 'none'>"}

Run every candidate model, including the frontier API baseline and its Arabic-prompted variant, on the identical set with the identical retrieval context if you use RAG. Report the mean total, the per-criterion breakdown, and the fatal-error rate. The per-criterion breakdown matters more than the total: a model that loses two points per item entirely on dialect handling might be fixable with prompt work, while one that loses them on grounding needs retrieval surgery, and one that loses them on correctness needs to be dropped.

For dialect specifically, published per-dialect evaluations such as DialectalArabicMMLU show how much scores move between MSA and regional varieties, and TII reports AraDice dialect sub-scores for the Falcon-H1-Arabic family. Your own dialect slice settles the question for your dialect, which is the only one that matters.

The second thing that transfers: tokenization economics

Arabic text costs more tokens than the equivalent English text on English-centric tokenizers, and that premium appears in four places at once: your API bill, time to first token, KV-cache memory and the number of concurrent Arabic sessions a GPU can hold.

The peer-reviewed evidence is blunt. Petrov and colleagues showed that the GPT-4 tokenizer encodes the same content in Standard Arabic with roughly 3.04 times as many tokens as English, and the ChatGPT tokenizer with about 4.40 times as many, with direct consequences for cost, latency and effective context length (arXiv:2305.15425). The original Jais team measured the same pattern at the word level: GPT-2's vocabulary fragments Arabic into about 4.17 tokens per Arabic word against roughly 1.1 per English word, while the bilingual Jais vocabulary achieves about 1.05 (fertility figures summarised here). Newer multilingual vocabularies have narrowed the gap, but a 1.5 to 2 times premium on an Arabic-inefficient tokenizer is still normal, and nothing on your invoice labels it.

Measure it on your own traffic before believing any ratio. Ten lines of Python gives you fertility, tokens per word, on a sample of your real conversations:

from transformers import AutoTokenizer

text = open("sample_arabic_conversations.txt", encoding="utf-8").read()
words = len(text.split())

# Verified repo IDs; add the other candidates from their model cards
for repo in ["QCRI/Fanar-2-27B-Instruct",
             "humain-ai/ALLaM-7B-Instruct-preview"]:
    tok = AutoTokenizer.from_pretrained(repo)
    fertility = len(tok(text).input_ids) / words
    print(f"{repo}: {fertility:.2f} tokens per Arabic word")

One caution: studies count words and normalise text differently, so compare ratios within your own sample rather than against published absolutes. Your sample is the number that will hit your bill.

Worked example: what the premium actually costs

Take a 400-word Arabic support answer and compare a model whose tokenizer gives 2.4 tokens per Arabic word against an Arabic-optimised model at 1.5. Same content, same job.

Spend. The answer is 960 tokens on the first tokenizer, 600 on the second, a 1.6 times premium on every request. At an illustrative $5 per million output tokens, one million such answers costs $4,800 versus $3,000, a $1,800 difference before you have written a word of prompt engineering. Multiply by your volume. And note the trap when comparing providers: a model priced 30% lower per token that fragments Arabic 1.6 times more is not 30% cheaper for you, it is roughly 12% more expensive.

Latency. Prefill is compute-bound on prompt length, so time to first token scales with the token count of your system prompt plus retrieved context. Decode is per token, so an answer fragmented into 1.6 times the tokens streams 1.6 times longer at the same hardware speed. Prompt tuning cannot remove this; it is in the vocabulary.

KV cache and concurrency. Per-token KV-cache size for a 7B-class model with 32 layers, 8 grouped-query KV heads and 128-dimension heads at fp16 is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. A 4,000-token Arabic conversation at the efficient tokenizer holds about 500 MB of KV per session; the same content at 1.6 times fertility holds about 800 MB. On a GPU with 60 GB free after model weights, that is roughly 120 concurrent sessions versus 75. Tokenization, not parameter count, decided your concurrency.

Context. A 256K context window (Falcon-H1-Arabic 7B and 34B) holds far more Arabic at a low fertility than a longer window holds at a high one. This is why the Jais 2 team built a custom 150,272-entry Arabic-centric vocabulary and report it makes inference efficient: the efficiency is not marketing, it is arithmetic you can reproduce with the snippet above.

The decision rule

Now the choice reduces to one comparison on one spreadsheet.

1. Run all candidates on your eval set. Suppose the frontier API scores 9.1 out of 10 and Falcon-H1-Arabic 7B scores 8.6. The gap is 0.5 points.
2. Set your tolerance in advance, in the currency of your use case: acceptable fatal-error rate on billing questions, acceptable dialect-mishandling rate. Do not set it after seeing the scores.
3. If the open model is inside your tolerance: run it on your own GPUs. You keep every conversation in-country, you pay a fixed infrastructure cost instead of a token bill that scales with Arabic verbosity, and you are not exposed to an API being retired (Jais 2's hosted endpoint went away in August 2026; its weights did not). A regional open model that scores within tolerance on your data is strictly the better enterprise choice: cheaper per request, in-country data, and no dependency on someone else's pricing page.
4. If the gap exceeds your tolerance: do not pay for it blindly. Quantify the frontier model's advantage in cost per resolved request: with the 1.6 times tokenization premium on an English-centric tokenizer and your ticket volume, what is the API bill for the extra 0.5 points? Then price the alternative: the same eval gap might close with a better Arabic system prompt, stronger retrieval into your policy documents, or three days of fine-tuning the open model on 2,000 of your own tickets. Very often the fine-tune is cheaper in month four than the API premium is in month one.

A practical note on keeping this sustainable: whatever prompts and gold answers survive the eval, version them. The Arabic system prompts that work, the dialect-handling instructions, the few-shot examples, these are assets, and they are exactly the kind of asset that walks out the door in a Markdown file. We keep them in a shared, versioned prompt library (PromptForge), and when we build and run these assistants for clients, from eval set through deployment, that is our AI engineering work, done by our own team here in Dubai.

What to do next

1. This week: pull 300 real items using the mix above. It is three days of one analyst and one domain expert, and it is reusable for every model you will ever evaluate.
2. In parallel: run the fertility snippet on 100 of your real conversations for every shortlisted model, including the frontier API's tokenizer. You now have the two numbers that transfer.
3. Next week: run all candidates through the judge prompt, with a human agreement check on 50 overlapping items. Build the table: eval score, fatal-error rate, tokens per request, cost per resolved request.
4. Then decide with the rule above. If the regional open model is inside tolerance, it wins on data residency and unit economics, and GCC regulators and your security team will thank you. If it is not, you now know exactly what the gap costs per month, which turns a vague preference into a negotiation.

The best Arabic LLM for enterprise is not a fixed answer, because "enterprise" is not a fixed workload. It is whichever of Falcon-H1-Arabic, ALLaM, Fanar 2, Jais 2 or a frontier API passes an eval set built from your own tickets, at a token cost you have measured rather than assumed. Both measurements take a week. The alternative is guessing, and guessing is the most expensive part of any AI project.

Originally published on Azrty.