跳到正文
Hugging Face Blog·· 2025-09-09精選AI 評分61

mmBERT 基於 ModernBERT 推出,訓練納入 1,833 種語言

mmBERT: ModernBERT goes Multilingual

AI 導讀

mmBERT 以 ModernBERT 為基礎推出,使用超過 3T tokens 的多語言文本訓練,並在最後階段納入 FineWeb2 的 1,833 種語言。

推薦理由

mmBERT 展示分階段擴充語言、逐步降低遮罩率的訓練設計,並報告在最後 100B tokens 才納入的低資源語言上取得明顯提升。

正文 · 繁體中文

簡而言之

本文介紹 mmBERT,這是一個最先進的大規模多語言編碼器模型,以超過 3T 個文字 token 訓練,涵蓋 1,800 多種語言。相較以往的多語言模型,它在效能和速度方面均有顯著提升,是首個超越 XLM-R 的模型,同時亦開發出有效學習低資源語言的新策略。mmBERT 以 ModernBERT 為基礎,採用極高速架構,並加入創新元件以高效進行多語言學習。

如想親自試用這些模型,本文末段提供了一些範例程式碼!

訓練數據

Distribution of each phase of pre-training
圖 1:訓練數據會逐步調整,加入更多語言,並在整個訓練過程中採用更平均的抽樣方式。

mmBERT 使用經仔細整理的多語言數據集進行訓練,總計超過 3T 個 token,並分為三個不同訓練階段。我們的訓練數據以三個主要的開源高質素網頁爬取數據集為基礎,兼顧多語言覆蓋範圍及數據質素:

DCLM 和篩選後的 DCLM提供目前最高質素的英文內容,作為強大英文表現的基礎(篩選後的數據來自 Dolmino)。此數據集採用最先進的網頁篩選技術,是訓練的重要組成部分。由於數據質素高,我們使用的英文比例遠高於上一代多語言編碼器模型(最高達 18%)。

FineWeb2提供廣泛的多語言網頁內容,涵蓋超過 1,800 種語言。此數據集讓我們在維持合理質素標準的同時,廣泛涵蓋不同語系和文字系統。

FineWeb2-HQ是 FineWeb2 的篩選子集,聚焦 20 種高資源語言。這個篩選版本提供質素更高的多語言內容,填補只含英文的篩選數據與廣泛多語言數據之間的差距。

訓練數據亦加入來自 Dolma、MegaWika v2、ProLong 等來源的專門語料:程式碼儲存庫(StarCoder、ProLong)、學術內容(ArXiv、PeS2o)、參考資料(Wikipedia、教科書)、社羣討論(StackExchange),以及指示遵循和數學數據集。

數據策略的關鍵創新是逐步納入語言策略(如圖 1所示)。在每個階段,我們會逐步從較平坦(即更接近均勻)的分佈抽樣,同時加入新語言。這表示,俄語等高資源語言在開始時佔數據比例較高(即 9%),到訓練最後階段則降至約一半。我們先在預訓練階段使用 60 種高資源語言,中期訓練擴展至 110 種語言,最後在衰減階段納入 FineWeb2 的全部 1,833 種語言。這讓我們能在避免重複過多的同時,充分運用有限的低資源語言數據,並維持整體數據質素。

訓練方案與創新元件

mmBERT 以 ModernBERT 架構為基礎,但加入了多項多語言學習創新:

架構

我們採用與 ModernBERT-base 相同的核心架構,包含 22 層和 1152 個中間維度,但改用 Gemma 2 tokenizer,以更妥善處理多語言文字。基礎模型有 110M 個非嵌入參數(由於詞彙表較大,總計 307M);小型版本則有 42M 個非嵌入參數(總計 140M)。

三階段訓練方法

我們採用經仔細設計的三階段訓練排程:

  1. 預訓練(2.3T 個 token):使用 60 種語言及 30% 遮罩率,進行暖身及穩定學習率階段
  2. 中期訓練(600B 個 token):將上下文擴展至 8192 個 token,採用更高質素數據,語言擴展至 110 種,遮罩率為 15%
  3. 衰減階段(100B 個 token):以反平方根方式衰減學習率,納入全部 1,833 種語言,遮罩率為 5%

創新訓練技術

反向遮罩比例排程:我們不採用固定遮罩率,而是在不同訓練階段逐步將遮罩比例由 30% 降至 15%,再降至 5%。這讓模型先以較高遮罩率學習基本表示,再以較低遮罩率專注理解更細緻的內容。

退火式語言學習:我們會動態調整多語言數據抽樣的温度,由 τ=0.7 降至 0.5,再降至 0.3。這個過程會由偏重高資源語言逐步轉向更平均的抽樣,讓模型先建立穩固的多語言基礎,再學習低資源語言。

逐步加入語言:我們不會同時使用所有語言訓練,而是在每個階段策略性地加入語言(60 → 110 → 1,833)。這既可避免有限的低資源數據被過度重複使用,提升學習效率,又能維持出色效能。

模型合併:衰減階段會訓練三個不同版本(以英文為主、涵蓋 110 種語言,以及涵蓋所有語言),再使用 TIES 合併整合各自優點,形成最終模型。

結果

自然語言理解(NLU)

GLUE performance
表 1:GLUE(英文)效能

英文效能:在英文 GLUE 基準測試(表 1)中,mmBERT base 表現出色,大幅超越 XLM-R(多語言 RoBERTa)base 和 mGTE base 等其他多語言模型;即使 mmBERT 訓練數據中英文比例不足 25%,表現仍可媲美純英文模型。

XTREME performance
表 2:XTREME(多語言)效能

多語言效能:如表 2所示,相較 XLM-R,mmBERT 在 XTREME 基準測試有顯著提升。值得注意的進步包括 XNLI 分類表現出色、大幅改善 TyDiQA 等問答任務,以及在 PAWS-X 和 XCOPA 跨語言理解測試中取得具競爭力的結果。

模型在大部分類別中表現良好,但在 NER 和 POS 標註等結構化預測任務中稍有例外,可能是 tokenizer 差異影響了詞語邊界判定。在這些類別中,模型表現與上一代大致相若,但可應用於更多語言。

檢索效能

MTEB v2 Eng performance
表 3:MTEB v2 英文效能

英文檢索:雖然 mmBERT 專為大規模多語言用途而設,但在 MTEB v2 英文基準測試(表 3)中,它不但大幅超越以往的多語言模型,表現甚至媲美 ModernBERT 等純英文模型!

MTEB v2 Multilingual performance
表 4:MTEB v2 多語言效能

多語言檢索:與其他模型相比,mmBERT 在 MTEB v2 多語言基準測試中持續取得更佳成績(表 4)。

CoIR performance
表 5:CoIR 程式碼基準測試效能

程式碼檢索:得益於以 Gemma 2 為基礎的新式 tokenizer,mmBERT 在程式碼方面亦表現出色(表 5),適用於各類文字數據。唯一表現更佳的模型是 EuroBERT,它可使用未公開開放的 Stack v2 數據集。

在衰減階段學習語言

mmBERT 最重要的創新特色之一,是證明低資源語言可在短暫的訓練衰減階段有效學習。我們透過只在最後 100B token 衰減階段才加入的語言進行測試,驗證了這種方法。

Improvements from adding new languages in the decay phase
圖 2:在衰減階段加入 1,700 多種語言可快速學習,而且透過模型合併後仍能保留這些成果。

效能大幅提升:如圖 2所示,在 TiQuaD(提格利尼亞語)和 FoQA(法羅語)測試中,衰減階段納入這些語言後,模型效能顯著改善。結果證明,逐步學習語言的方法十分有效。

媲美大型模型:mmBERT 只在最後訓練階段接觸這些語言,效能仍超越規模大得多的模型。在曾有大型語言模型參與基準測試的法羅語問答任務中,mmBERT 的表現勝過 Google Gemini 2.5 Pro 和 OpenAI o3。

快速學習機制:衰減階段的語言學習之所以成功,是因為模型可善用早期階段建立的穩固多語言基礎。接觸新語言時,模型能迅速調整既有的跨語言表示,而非從零開始學習。

模型合併的優點:最終 mmBERT 模型透過 TIES 合併,成功保留衰減階段大部分改進,同時受惠於以英文為主及高資源語言版本的優點。

效率提升

透過繼承自 ModernBERT 的架構改進,mmBERT 相較上一代多語言編碼器模型,大幅提升效率:

mmBERT is much more efficient than previous multilingual models
圖 3:mmBERT 的效率明顯高於以往的多語言模型,最高可達 2–4 倍!

吞吐量效能:如圖 3所示,無論序列長度如何,mmBERT 處理文字的速度都比現有多語言模型快得多。小型和基礎模型均較以往的多語言編碼器大幅提速。

現代架構的優點:效率提升主要來自兩項技術改進:

  • Flash Attention 2:最佳化注意力計算,提升記憶體使用效率和速度
  • 取消填充技術:處理過程中移除不必要的填充 token

序列長度擴展:與舊模型最多隻支援 512 個 token 不同,mmBERT 可高效處理最多 8,192 個 token,同時維持高吞吐量。這令它適合處理愈來愈常見的多語言長文件。

能源效益:吞吐量提升配合現代架構,降低了推理的運算成本,讓 mmBERT 更適合在需要大規模多語言支援的正式環境中部署。

這些效率改進不但令 mmBERT 比以往的多語言編碼器更準確,也大幅提高其實際應用價值。

使用範例

只需幾行程式碼,就能使用這些模型!

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmBERT-base")

def predict_masked_token(text):
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
    predictions = outputs.logits[mask_indices]
    top_tokens, top_indices = torch.topk(predictions, 5, dim=-1)
    return [tokenizer.decode(token) for token in top_indices[0]]

# Works across languages
texts = [
    "The capital of France is <mask>.",
    "La capital de España es <mask>.",
    "Die Hauptstadt von Deutschland ist <mask>.",
]

for text in texts:
    predictions = predict_masked_token(text)
    print(f"Text: {text}")
    print(f"Predictions: {predictions}\n")

微調範例

編碼器

按此查看如何使用 Sentence Transformers,將模型微調為密集嵌入模型

import argparse

from datasets import load_dataset
from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
)
from sentence_transformers.evaluation import TripletEvaluator
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
from sentence_transformers.training_args import BatchSamplers

def main():
    # parse the lr & model name
    parser = argparse.ArgumentParser()
    parser.add_argument("--lr", type=float, default=8e-5)
    parser.add_argument("--model_name", type=str, default="jhu-clsp/mmBERT-small")
    args = parser.parse_args()
    lr = args.lr
    model_name = args.model_name
    model_shortname = model_name.split("/")[-1]

    # 1. Load a model to finetune
    model = SentenceTransformer(model_name)

    # 2. Load a dataset to finetune on
    dataset = load_dataset(
        "sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1",
        "triplet-hard",
        split="train",
    )
    dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)
    train_dataset = dataset_dict["train"].select(range(1_250_000))
    eval_dataset = dataset_dict["test"]

    # 3. Define a loss function
    loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16)  # Increase mini_batch_size if you have enough VRAM

    run_name = f"{model_shortname}-DPR-{lr}"
    # 4. (Optional) Specify training arguments
    args = SentenceTransformerTrainingArguments(
        # Required parameter:
        output_dir=f"output/{model_shortname}/{run_name}",
        # Optional training parameters:
        num_train_epochs=1,
        per_device_train_batch_size=512,
        per_device_eval_batch_size=512,
        warmup_ratio=0.05,
        fp16=False,  # Set to False if GPU can't handle FP16
        bf16=True,  # Set to True if GPU supports BF16
        batch_sampler=BatchSamplers.NO_DUPLICATES,  # (Cached)MultipleNegativesRankingLoss benefits from no duplicates
        learning_rate=lr,
        # Optional tracking/debugging parameters:
        save_strategy="steps",
        save_steps=500,
        save_total_limit=2,
        logging_steps=500,
        run_name=run_name,  # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed
    )

    # 5. (Optional) Create an evaluator & evaluate the base model
    dev_evaluator = TripletEvaluator(
        anchors=eval_dataset["query"],
        positives=eval_dataset["positive"],
        negatives=eval_dataset["negative"],
        name="msmarco-co-condenser-dev",
    )
    dev_evaluator(model)

    # 6. Create a trainer & train
    trainer = SentenceTransformerTrainer(
        model=model,
        args=args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        loss=loss,
        evaluator=dev_evaluator,
    )
    trainer.train()

    # 7. (Optional) Evaluate the trained model on the evaluator after training
    dev_evaluator(model)

    # 8. Save the model
    model.save_pretrained(f"output/{model_shortname}/{run_name}/final")

    # 9. (Optional) Push it to the Hugging Face Hub
    model.push_to_hub(run_name, private=False)

if __name__ == "__main__":
    main()

按此查看如何使用 PyLate,將模型微調為多向量嵌入模型

from datasets import load_dataset
from pylate import losses, models, utils
from sentence_transformers import (
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
)

def main():
    # Load the datasets required for knowledge distillation (train, queries, documents)
    train = load_dataset(
        path="lightonai/ms-marco-en-bge",
        name="train",
    )

    queries = load_dataset(
        path="lightonai/ms-marco-en-bge",
        name="queries",
    )

    documents = load_dataset(
        path="lightonai/ms-marco-en-bge",
        name="documents",
    )

    # Set the transformation to load the documents/queries texts using the corresponding ids on the fly
    train.set_transform(
        utils.KDProcessing(queries=queries, documents=documents).transform,
    )

    # Define the base model, training parameters, and output directory
    num_train_epochs = 1
    lr = 8e-5
    batch_size = 16
    accum_steps = 1
    model_name = "jhu-clsp/mmBERT-small"
    model_shortname = model_name.split("/")[-1]

    # Set the run name for logging and output directory
    run_name = f"{model_shortname}-colbert-KD-{lr}"
    output_dir = f"output/{model_shortname}/{run_name}"

    # Initialize the ColBERT model from the base model
    model = models.ColBERT(model_name_or_path=model_name)

    # Configure the training arguments (e.g., epochs, batch size, learning rate)
    args = SentenceTransformerTrainingArguments(
        output_dir=output_dir,
        num_train_epochs=num_train_epochs,
        per_device_train_batch_size=batch_size,
        fp16=False,  # Set to False if you get an error that your GPU can't run on FP16
        bf16=True,  # Set to True if you have a GPU that supports BF16
        run_name=run_name,
        logging_steps=10,
        learning_rate=lr,
        gradient_accumulation_steps=accum_steps,
        warmup_ratio=0.05,
    )

    # Use the Distillation loss function for training
    train_loss = losses.Distillation(model=model)

    # Initialize the trainer
    trainer = SentenceTransformerTrainer(
        model=model,
        args=args,
        train_dataset=train,
        loss=train_loss,
        data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),
    )

    # Start the training process
    trainer.train()

    model.save_pretrained(f"{output_dir}/final")

if __name__ == "__main__":
    main()

按此查看如何使用 Sentence Transformers,將模型微調為稀疏檢索模型

import logging

from datasets import load_dataset

from sentence_transformers import (
    SparseEncoder,
    SparseEncoderModelCardData,
    SparseEncoderTrainer,
    SparseEncoderTrainingArguments,
)
from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
from sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss
from sentence_transformers.training_args import BatchSamplers

logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)

# 1. Load a model to finetune with 2. (Optional) model card data
model = SparseEncoder(
    "jhu-clsp/mmBERT-small",
    model_card_data=SparseEncoderModelCardData(
        language="en",
        license="apache-2.0",
    )
)

# 3. Load a dataset to finetune on
full_dataset = load_dataset("sentence-transformers/natural-questions", split="train").select(range(100_000))
dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"]
eval_dataset = dataset_dict["test"]

# 4. Define a loss function
loss = SpladeLoss(
    model=model,
    loss=SparseMultipleNegativesRankingLoss(model=model),
    query_regularizer_weight=5e-5,
    document_regularizer_weight=3e-5,
)

# 5. (Optional) Specify training arguments
run_name = "splade-distilbert-base-uncased-nq"
args = SparseEncoderTrainingArguments(
    # Required parameter:
    output_dir=f"models/{run_name}",
    # Optional training parameters:
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    fp16=True,  # Set to False if you get an error that your GPU can't run on FP16
    bf16=False,  # Set to True if you have a GPU that supports BF16
    batch_sampler=BatchSamplers.NO_DUPLICATES,  # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
    # Optional tracking/debugging parameters:
    eval_strategy="steps",
    eval_steps=1000,
    save_strategy="steps",
    save_steps=1000,
    save_total_limit=2,
    logging_steps=200,
    run_name=run_name,  # Will be used in W&B if `wandb` is installed
)

# 6. (Optional) Create an evaluator & evaluate the base model
dev_evaluator = SparseNanoBEIREvaluator(dataset_names=["msmarco", "nfcorpus", "nq"], batch_size=16)

# 7. Create a trainer & train
trainer = SparseEncoderTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    loss=loss,
    evaluator=dev_evaluator,
)
trainer.train()

# 8. Evaluate the model performance again after training
dev_evaluator(model)

# 9. Save the trained model
model.save_pretrained(f"models/{run_name}/final")

# 10. (Optional) Push it to the Hugging Face Hub
model.push_to_hub(run_name)

按此查看如何使用 Sentence Transformers,將模型微調為 reranker 模型

import logging
import traceback

import torch
from datasets import load_dataset

from sentence_transformers import SentenceTransformer
from sentence_transformers.cross_encoder import (
    CrossEncoder,
    CrossEncoderModelCardData,
    CrossEncoderTrainer,
    CrossEncoderTrainingArguments,
)
from sentence_transformers.cross_encoder.evaluation import (
    CrossEncoderNanoBEIREvaluator,
    CrossEncoderRerankingEvaluator,
)
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
from sentence_transformers.evaluation import SequentialEvaluator
from sentence_transformers.util import mine_hard_negatives

# Set the log level to INFO to get more information
logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)


def main():
    model_name = "jhu-clsp/mmBERT-small"

    train_batch_size = 64
    num_epochs = 1
    num_hard_negatives = 5  # How many hard negatives should be mined for each question-answer pair

    # 1a. Load a model to finetune with 1b. (Optional) model card data
    model = CrossEncoder(
        model_name,
        model_card_data=CrossEncoderModelCardData(
            language="en",
            license="apache-2.0",
        ),
    )
    print("Model max length:", model.max_length)
    print("Model num labels:", model.num_labels)

    # 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq
    logging.info("Read the gooaq training dataset")
    full_dataset = load_dataset("sentence-transformers/gooaq", split="train").select(range(100_000))
    dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
    train_dataset = dataset_dict["train"]
    eval_dataset = dataset_dict["test"]
    logging.info(train_dataset)
    logging.info(eval_dataset)

    # 2b. Modify our training dataset to include hard negatives using a very efficient embedding model
    embedding_model = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1", device="cpu")
    hard_train_dataset = mine_hard_negatives(
        train_dataset,
        embedding_model,
        num_negatives=num_hard_negatives,  # How many negatives per question-answer pair
        margin=0,  # Similarity between query and negative samples should be x lower than query-positive similarity
        range_min=0,  # Skip the x most similar samples
        range_max=100,  # Consider only the x most similar samples
        sampling_strategy="top",  # Sample the top negatives from the range
        batch_size=4096,  # Use a batch size of 4096 for the embedding model
        output_format="labeled-pair",  # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss
        use_faiss=True,
    )
    logging.info(hard_train_dataset)

    # 2c. (Optionally) Save the hard training dataset to disk
    # hard_train_dataset.save_to_disk("gooaq-hard-train")
    # Load again with:
    # hard_train_dataset = load_from_disk("gooaq-hard-train")

    # 3. Define our training loss.
    # pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`
    loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))

    # 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking
    nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(
        dataset_names=["msmarco", "nfcorpus", "nq"],
        batch_size=train_batch_size,
    )

    # 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs
    # We include the positive answer in the list of negatives, so the evaluator can use the performance of the
    # embedding model as a baseline.
    hard_eval_dataset = mine_hard_negatives(
        eval_dataset,
        embedding_model,
        corpus=full_dataset["answer"],  # Use the full dataset as the corpus
        num_negatives=30,  # How many documents to rerank
        batch_size=4096,
        include_positives=True,
        output_format="n-tuple",
        use_faiss=True,
    )
    logging.info(hard_eval_dataset)
    reranking_evaluator = CrossEncoderRerankingEvaluator(
        samples=[
            {
                "query": sample["question"],
                "positive": [sample["answer"]],
                "documents": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],
            }
            for sample in hard_eval_dataset
        ],
        batch_size=train_batch_size,
        name="gooaq-dev",
        # Realistic setting: only rerank the positives that the retriever found
        # Set to True to rerank *all* positives
        always_rerank_positives=False,
    )

    # 4c. Combine the evaluators & run the base model on them
    evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])
    evaluator(model)

    # 5. Define the training arguments
    short_model_name = model_name if "/" not in model_name else model_name.split("/")[-1]
    run_name = f"reranker-{short_model_name}-gooaq-bce"
    args = CrossEncoderTrainingArguments(
        # Required parameter:
        output_dir=f"models/{run_name}",
        # Optional training parameters:
        num_train_epochs=num_epochs,
        per_device_train_batch_size=train_batch_size,
        per_device_eval_batch_size=train_batch_size,
        learning_rate=2e-5,
        warmup_ratio=0.1,
        fp16=False,  # Set to False if you get an error that your GPU can't run on FP16
        bf16=True,  # Set to True if you have a GPU that supports BF16
        dataloader_num_workers=4,
        load_best_model_at_end=True,
        metric_for_best_model="eval_gooaq-dev_ndcg@10",
        # Optional tracking/debugging parameters:
        eval_strategy="steps",
        eval_steps=1000,
        save_strategy="steps",
        save_steps=1000,
        save_total_limit=2,
        logging_steps=200,
        logging_first_step=True,
        run_name=run_name,  # Will be used in W&B if `wandb` is installed
        seed=12,
    )

    # 6. Create the trainer & start training
    trainer = CrossEncoderTrainer(
        model=model,
        args=args,
        train_dataset=hard_train_dataset,
        loss=loss,
        evaluator=evaluator,
    )
    trainer.train()

    # 7. Evaluate the final model, useful to include these in the model card
    evaluator(model)

    # 8. Save the final model
    final_output_dir = f"models/{run_name}/final"
    model.save_pretrained(final_output_dir)

    # 9. (Optional) save the model to the Hugging Face Hub!
    # It is recommended to run `huggingface-cli login` to log into your Hugging Face account first
    try:
        model.push_to_hub(run_name)
    except Exception:
        logging.error(
            f"Error uploading model to the Hugging Face Hub:\n{traceback.format_exc()}To upload it manually, you can run "
            f"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` "
            f"and saving it using `model.push_to_hub('{run_name}')`."
        )


if __name__ == "__main__":
    main()

模型系列與連結

標準模型:

  • mmBERT-small(總計 140M 個參數,42M 個非嵌入參數)
  • mmBERT-base(總計 307M 個參數,110M 個非嵌入參數)

研究資源:

來源:Hugging Face Blog · huggingface.co