跳到正文
DEV Community· Japan Open Data Lab·· 9 小時前AI 評分45

ja-nli-v1:連接生成與信任的日語 NLI 檢查器(封閉測試 0.961,零樣本基線 0.79–0.81)

A Japanese NLI checker for the space between generation and trust (sealed 0.961, zero-shot baselines 0.79–0.81)

AI 導讀

ja-nli-v1 是一款小型日語自然語言推理模型,根據前提與假設判斷藴涵、中立或矛盾。模型在 5,000 組封閉測試中取得 0.9612 準確率及 Macro F1,兩個多語言模型的零樣本分數為 0.7920 和 0.8094。模型採 Apache 2.0 授權並附有 ONNX,CPU 每組約需 109 ms;測試集由合成資料衍生,實際文本表現預期較低。

正文

Japan Open Data Lab

TL;DR — We released ja-nli-v1, a small Japanese natural language inference model: given a premise and a hypothesis, it returns probabilities for entailment / neutral / contradiction. On a sealed test of 5,000 pairs it scores accuracy and Macro F1 of 0.9612; on a 600-pair human-verified set, 0.9567, with entailment↔contradiction confusions at 3 (0.5%). Two multilingual NLI models used zero-shot score 0.79–0.81 on the same sealed set. A premise-shuffle diagnostic confirms the model actually reads the premise: swap in an unrelated one and accuracy falls to ~0.38. Trained without any existing NLI dataset — licensing, not preference. Apache 2.0, ONNX included, ~109 ms per pair on CPU.

Model page (try the demo from there): https://huggingface.co/japan-opendata/ja-nli-v1
Browser demo: https://huggingface.co/spaces/japan-opendata/ja-nli-demo

  1. A verification part, meant to sit next to generation. Does the RAG answer contradict its source passage? Does the claim match the search result it cites? Those are premise–hypothesis questions, and in Japanese they usually get answered by a large multilingual model or not at all. This is the small, local, CPU-priced version.
  2. The sanity checks are the feature. Shuffle the premise for an unrelated one and accuracy drops to ~0.38, with most outputs collapsing to neutral — the model is lost without the premise, which is what you want. A hypothesis-only probe (can it guess the label from the hypothesis alone?) was driven down to ~0.63 on the human-verified set by diversifying the training data, so the shortcut is weak. An NLI model that doesn't read its premise is a label guesser; we measured that ours does.
  3. No existing NLI data was trained on. JSNLI, JNLI, JSICK, JaNLI, XNLI, ANLI — all excluded on license terms. The training pairs are synthetic, built from seed corpora limited to public-domain-licensed government documents (PDL 1.0) and MIT / Apache-2.0 Japanese OSS documentation, generated by an open-weight LLM and blind-judged by a different model. The zero-shot baselines above were measured for reference only.

Use it

from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "japan-opendata/ja-nli-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()
# Give it a premise and a hypothesis; it returns probabilities for
# entailment / neutral / contradiction (mapping in inference_config.json).
# Max length 1,024 tokens; truncation applies to the premise side only.
# Full reference implementation (PyTorch and ONNX): examples/infer_nli.py

The ONNX build (fp32, opset 17) matched PyTorch predictions exactly on 200 development and 200 sealed items. Speed: ~109 ms per pair on a Core i7-11700F, measured on inputs that include longer premises. No INT8 build is shipped.

Measured results

Sealed test (5,000 pairs, separated from training, single final run):

  • Accuracy 0.9612, Macro F1 0.9612
  • Zero-shot comparison, same conditions: two multilingual mDeBERTa-family NLI models at 0.7920 and 0.8094

Human-verified evaluation (600 pairs):

  • Accuracy 0.9567; entailment↔contradiction confusions: 3 (0.5%)
  • The same zero-shot models: 0.8433 each

Diagnostics:

  • Premise shuffled to an unrelated text: accuracy falls to ~0.38, most predictions collapse to neutral
  • Hypothesis-only probe: ~0.63 on the human-verified set (kept low by design)

Training data

Seed corpora: PDL 1.0 public documents, and Japanese OSS documentation with confirmed MIT / Apache-2.0 licenses — nothing else. Pairs were generated by an open-weight LLM and verified by blind judgment from a separate model. A random 200-pair audit found label validity 95.0% and Japanese naturalness 94.5%.

What didn't work / Limitations

  • The evaluation ground is, again, ours. Both test sets are synthetic-derived, built and labeled under our pipeline. The zero-shot models were graded on Japanese pairs written to our distribution; on the standard Japanese NLI benchmarks they were built for, the comparison would look different. We did not train on those benchmarks, so we also cannot report on them as a neutral party — treat 0.9612 as performance on verified synthetic pairs, with real-world text expected to score lower.
  • The hypothesis-only probe is weakened, not dead. ~0.63 is well below the full model's 0.9567, but above the 0.33 chance level — some hypothesis-side signal remains, as it does in essentially every NLI dataset ever built, including the famous ones. The premise-shuffle collapse (~0.38) is the stronger evidence that the premise drives the decision.
  • ~109 ms is the slowest in our series so far. Premise-plus-hypothesis pairs with long premises cost more than single short texts. Fine for checking generated answers; budget accordingly for bulk scoring.
  • Truncation cuts the premise, never the hypothesis. Past 1,024 tokens, the end of a long source passage silently stops participating. For long-document checks, chunk the premise yourself rather than letting the tokenizer decide what matters.

This is the eleventh in our line of small, single-purpose Japanese models. NLI is the one where "the model got a high score" is least trustworthy on its own — so this release leads with the diagnostics that try to catch the model cheating, and publishes what they found.

If you check generated Japanese text against its sources today: dedicated NLI model, a prompt to a big LLM, or nothing yet — and which failure worries you most, a missed contradiction or a false alarm?

Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).

來源:DEV Community · dev.to