mmBERT 基於 ModernBERT 推出,訓練納入 1,833 種語言
mmBERT: ModernBERT goes Multilingual
mmBERT 以 ModernBERT 為基礎推出,使用超過 3T tokens 的多語言文本訓練,並在最後階段納入 FineWeb2 的 1,833 種語言。
mmBERT 展示分階段擴充語言、逐步降低遮罩率的訓練設計,並報告在最後 100B tokens 才納入的低資源語言上取得明顯提升。
簡而言之
本文介紹 mmBERT,這是一個最先進的大規模多語言編碼器模型,以超過 3T 個文字 token 訓練,涵蓋 1,800 多種語言。相較以往的多語言模型,它在效能和速度方面均有顯著提升,是首個超越 XLM-R 的模型,同時亦開發出有效學習低資源語言的新策略。mmBERT 以 ModernBERT 為基礎,採用極高速架構,並加入創新元件以高效進行多語言學習。
如想親自試用這些模型,本文末段提供了一些範例程式碼!
訓練數據
mmBERT 使用經仔細整理的多語言數據集進行訓練,總計超過 3T 個 token,並分為三個不同訓練階段。我們的訓練數據以三個主要的開源高質素網頁爬取數據集為基礎,兼顧多語言覆蓋範圍及數據質素:
DCLM 和篩選後的 DCLM提供目前最高質素的英文內容,作為強大英文表現的基礎(篩選後的數據來自 Dolmino)。此數據集採用最先進的網頁篩選技術,是訓練的重要組成部分。由於數據質素高,我們使用的英文比例遠高於上一代多語言編碼器模型(最高達 18%)。
FineWeb2提供廣泛的多語言網頁內容,涵蓋超過 1,800 種語言。此數據集讓我們在維持合理質素標準的同時,廣泛涵蓋不同語系和文字系統。
FineWeb2-HQ是 FineWeb2 的篩選子集,聚焦 20 種高資源語言。這個篩選版本提供質素更高的多語言內容,填補只含英文的篩選數據與廣泛多語言數據之間的差距。
訓練數據亦加入來自 Dolma、MegaWika v2、ProLong 等來源的專門語料:程式碼儲存庫(StarCoder、ProLong)、學術內容(ArXiv、PeS2o)、參考資料(Wikipedia、教科書)、社羣討論(StackExchange),以及指示遵循和數學數據集。
數據策略的關鍵創新是逐步納入語言策略(如圖 1所示)。在每個階段,我們會逐步從較平坦(即更接近均勻)的分佈抽樣,同時加入新語言。這表示,俄語等高資源語言在開始時佔數據比例較高(即 9%),到訓練最後階段則降至約一半。我們先在預訓練階段使用 60 種高資源語言,中期訓練擴展至 110 種語言,最後在衰減階段納入 FineWeb2 的全部 1,833 種語言。這讓我們能在避免重複過多的同時,充分運用有限的低資源語言數據,並維持整體數據質素。
訓練方案與創新元件
mmBERT 以 ModernBERT 架構為基礎,但加入了多項多語言學習創新:
架構
我們採用與 ModernBERT-base 相同的核心架構,包含 22 層和 1152 個中間維度,但改用 Gemma 2 tokenizer,以更妥善處理多語言文字。基礎模型有 110M 個非嵌入參數(由於詞彙表較大,總計 307M);小型版本則有 42M 個非嵌入參數(總計 140M)。
三階段訓練方法
我們採用經仔細設計的三階段訓練排程:
- 預訓練(2.3T 個 token):使用 60 種語言及 30% 遮罩率,進行暖身及穩定學習率階段
- 中期訓練(600B 個 token):將上下文擴展至 8192 個 token,採用更高質素數據,語言擴展至 110 種,遮罩率為 15%
- 衰減階段(100B 個 token):以反平方根方式衰減學習率,納入全部 1,833 種語言,遮罩率為 5%
創新訓練技術
反向遮罩比例排程:我們不採用固定遮罩率,而是在不同訓練階段逐步將遮罩比例由 30% 降至 15%,再降至 5%。這讓模型先以較高遮罩率學習基本表示,再以較低遮罩率專注理解更細緻的內容。
退火式語言學習:我們會動態調整多語言數據抽樣的温度,由 τ=0.7 降至 0.5,再降至 0.3。這個過程會由偏重高資源語言逐步轉向更平均的抽樣,讓模型先建立穩固的多語言基礎,再學習低資源語言。
逐步加入語言:我們不會同時使用所有語言訓練,而是在每個階段策略性地加入語言(60 → 110 → 1,833)。這既可避免有限的低資源數據被過度重複使用,提升學習效率,又能維持出色效能。
模型合併:衰減階段會訓練三個不同版本(以英文為主、涵蓋 110 種語言,以及涵蓋所有語言),再使用 TIES 合併整合各自優點,形成最終模型。
結果
自然語言理解(NLU)
英文效能:在英文 GLUE 基準測試(表 1)中,mmBERT base 表現出色,大幅超越 XLM-R(多語言 RoBERTa)base 和 mGTE base 等其他多語言模型;即使 mmBERT 訓練數據中英文比例不足 25%,表現仍可媲美純英文模型。
多語言效能:如表 2所示,相較 XLM-R,mmBERT 在 XTREME 基準測試有顯著提升。值得注意的進步包括 XNLI 分類表現出色、大幅改善 TyDiQA 等問答任務,以及在 PAWS-X 和 XCOPA 跨語言理解測試中取得具競爭力的結果。
模型在大部分類別中表現良好,但在 NER 和 POS 標註等結構化預測任務中稍有例外,可能是 tokenizer 差異影響了詞語邊界判定。在這些類別中,模型表現與上一代大致相若,但可應用於更多語言。
檢索效能
英文檢索:雖然 mmBERT 專為大規模多語言用途而設,但在 MTEB v2 英文基準測試(表 3)中,它不但大幅超越以往的多語言模型,表現甚至媲美 ModernBERT 等純英文模型!
多語言檢索:與其他模型相比,mmBERT 在 MTEB v2 多語言基準測試中持續取得更佳成績(表 4)。
程式碼檢索:得益於以 Gemma 2 為基礎的新式 tokenizer,mmBERT 在程式碼方面亦表現出色(表 5),適用於各類文字數據。唯一表現更佳的模型是 EuroBERT,它可使用未公開開放的 Stack v2 數據集。
在衰減階段學習語言
mmBERT 最重要的創新特色之一,是證明低資源語言可在短暫的訓練衰減階段有效學習。我們透過只在最後 100B token 衰減階段才加入的語言進行測試,驗證了這種方法。
效能大幅提升:如圖 2所示,在 TiQuaD(提格利尼亞語)和 FoQA(法羅語)測試中,衰減階段納入這些語言後,模型效能顯著改善。結果證明,逐步學習語言的方法十分有效。
媲美大型模型:mmBERT 只在最後訓練階段接觸這些語言,效能仍超越規模大得多的模型。在曾有大型語言模型參與基準測試的法羅語問答任務中,mmBERT 的表現勝過 Google Gemini 2.5 Pro 和 OpenAI o3。
快速學習機制:衰減階段的語言學習之所以成功,是因為模型可善用早期階段建立的穩固多語言基礎。接觸新語言時,模型能迅速調整既有的跨語言表示,而非從零開始學習。
模型合併的優點:最終 mmBERT 模型透過 TIES 合併,成功保留衰減階段大部分改進,同時受惠於以英文為主及高資源語言版本的優點。
效率提升
透過繼承自 ModernBERT 的架構改進,mmBERT 相較上一代多語言編碼器模型,大幅提升效率:
吞吐量效能:如圖 3所示,無論序列長度如何,mmBERT 處理文字的速度都比現有多語言模型快得多。小型和基礎模型均較以往的多語言編碼器大幅提速。
現代架構的優點:效率提升主要來自兩項技術改進:
- Flash Attention 2:最佳化注意力計算,提升記憶體使用效率和速度
- 取消填充技術:處理過程中移除不必要的填充 token
序列長度擴展:與舊模型最多隻支援 512 個 token 不同,mmBERT 可高效處理最多 8,192 個 token,同時維持高吞吐量。這令它適合處理愈來愈常見的多語言長文件。
能源效益:吞吐量提升配合現代架構,降低了推理的運算成本,讓 mmBERT 更適合在需要大規模多語言支援的正式環境中部署。
這些效率改進不但令 mmBERT 比以往的多語言編碼器更準確,也大幅提高其實際應用價值。
使用範例
只需幾行程式碼,就能使用這些模型!
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmBERT-base")
def predict_masked_token(text):
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
predictions = outputs.logits[mask_indices]
top_tokens, top_indices = torch.topk(predictions, 5, dim=-1)
return [tokenizer.decode(token) for token in top_indices[0]]
# Works across languages
texts = [
"The capital of France is <mask>.",
"La capital de España es <mask>.",
"Die Hauptstadt von Deutschland ist <mask>.",
]
for text in texts:
predictions = predict_masked_token(text)
print(f"Text: {text}")
print(f"Predictions: {predictions}\n")
微調範例
編碼器
按此查看如何使用 Sentence Transformers,將模型微調為密集嵌入模型
import argparse
from datasets import load_dataset
from sentence_transformers import (
SentenceTransformer,
SentenceTransformerTrainer,
SentenceTransformerTrainingArguments,
)
from sentence_transformers.evaluation import TripletEvaluator
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
from sentence_transformers.training_args import BatchSamplers
def main():
# parse the lr & model name
parser = argparse.ArgumentParser()
parser.add_argument("--lr", type=float, default=8e-5)
parser.add_argument("--model_name", type=str, default="jhu-clsp/mmBERT-small")
args = parser.parse_args()
lr = args.lr
model_name = args.model_name
model_shortname = model_name.split("/")[-1]
# 1. Load a model to finetune
model = SentenceTransformer(model_name)
# 2. Load a dataset to finetune on
dataset = load_dataset(
"sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1",
"triplet-hard",
split="train",
)
dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"].select(range(1_250_000))
eval_dataset = dataset_dict["test"]
# 3. Define a loss function
loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16) # Increase mini_batch_size if you have enough VRAM
run_name = f"{model_shortname}-DPR-{lr}"
# 4. (Optional) Specify training arguments
args = SentenceTransformerTrainingArguments(
# Required parameter:
output_dir=f"output/{model_shortname}/{run_name}",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=512,
per_device_eval_batch_size=512,
warmup_ratio=0.05,
fp16=False, # Set to False if GPU can't handle FP16
bf16=True, # Set to True if GPU supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES, # (Cached)MultipleNegativesRankingLoss benefits from no duplicates
learning_rate=lr,
# Optional tracking/debugging parameters:
save_strategy="steps",
save_steps=500,
save_total_limit=2,
logging_steps=500,
run_name=run_name, # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed
)
# 5. (Optional) Create an evaluator & evaluate the base model
dev_evaluator = TripletEvaluator(
anchors=eval_dataset["query"],
positives=eval_dataset["positive"],
negatives=eval_dataset["negative"],
name="msmarco-co-condenser-dev",
)
dev_evaluator(model)
# 6. Create a trainer & train
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
evaluator=dev_evaluator,
)
trainer.train()
# 7. (Optional) Evaluate the trained model on the evaluator after training
dev_evaluator(model)
# 8. Save the model
model.save_pretrained(f"output/{model_shortname}/{run_name}/final")
# 9. (Optional) Push it to the Hugging Face Hub
model.push_to_hub(run_name, private=False)
if __name__ == "__main__":
main()
按此查看如何使用 PyLate,將模型微調為多向量嵌入模型
from datasets import load_dataset
from pylate import losses, models, utils
from sentence_transformers import (
SentenceTransformerTrainer,
SentenceTransformerTrainingArguments,
)
def main():
# Load the datasets required for knowledge distillation (train, queries, documents)
train = load_dataset(
path="lightonai/ms-marco-en-bge",
name="train",
)
queries = load_dataset(
path="lightonai/ms-marco-en-bge",
name="queries",
)
documents = load_dataset(
path="lightonai/ms-marco-en-bge",
name="documents",
)
# Set the transformation to load the documents/queries texts using the corresponding ids on the fly
train.set_transform(
utils.KDProcessing(queries=queries, documents=documents).transform,
)
# Define the base model, training parameters, and output directory
num_train_epochs = 1
lr = 8e-5
batch_size = 16
accum_steps = 1
model_name = "jhu-clsp/mmBERT-small"
model_shortname = model_name.split("/")[-1]
# Set the run name for logging and output directory
run_name = f"{model_shortname}-colbert-KD-{lr}"
output_dir = f"output/{model_shortname}/{run_name}"
# Initialize the ColBERT model from the base model
model = models.ColBERT(model_name_or_path=model_name)
# Configure the training arguments (e.g., epochs, batch size, learning rate)
args = SentenceTransformerTrainingArguments(
output_dir=output_dir,
num_train_epochs=num_train_epochs,
per_device_train_batch_size=batch_size,
fp16=False, # Set to False if you get an error that your GPU can't run on FP16
bf16=True, # Set to True if you have a GPU that supports BF16
run_name=run_name,
logging_steps=10,
learning_rate=lr,
gradient_accumulation_steps=accum_steps,
warmup_ratio=0.05,
)
# Use the Distillation loss function for training
train_loss = losses.Distillation(model=model)
# Initialize the trainer
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train,
loss=train_loss,
data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),
)
# Start the training process
trainer.train()
model.save_pretrained(f"{output_dir}/final")
if __name__ == "__main__":
main()
按此查看如何使用 Sentence Transformers,將模型微調為稀疏檢索模型
import logging
from datasets import load_dataset
from sentence_transformers import (
SparseEncoder,
SparseEncoderModelCardData,
SparseEncoderTrainer,
SparseEncoderTrainingArguments,
)
from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
from sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss
from sentence_transformers.training_args import BatchSamplers
logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
# 1. Load a model to finetune with 2. (Optional) model card data
model = SparseEncoder(
"jhu-clsp/mmBERT-small",
model_card_data=SparseEncoderModelCardData(
language="en",
license="apache-2.0",
)
)
# 3. Load a dataset to finetune on
full_dataset = load_dataset("sentence-transformers/natural-questions", split="train").select(range(100_000))
dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"]
eval_dataset = dataset_dict["test"]
# 4. Define a loss function
loss = SpladeLoss(
model=model,
loss=SparseMultipleNegativesRankingLoss(model=model),
query_regularizer_weight=5e-5,
document_regularizer_weight=3e-5,
)
# 5. (Optional) Specify training arguments
run_name = "splade-distilbert-base-uncased-nq"
args = SparseEncoderTrainingArguments(
# Required parameter:
output_dir=f"models/{run_name}",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
learning_rate=2e-5,
warmup_ratio=0.1,
fp16=True, # Set to False if you get an error that your GPU can't run on FP16
bf16=False, # Set to True if you have a GPU that supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES, # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
# Optional tracking/debugging parameters:
eval_strategy="steps",
eval_steps=1000,
save_strategy="steps",
save_steps=1000,
save_total_limit=2,
logging_steps=200,
run_name=run_name, # Will be used in W&B if `wandb` is installed
)
# 6. (Optional) Create an evaluator & evaluate the base model
dev_evaluator = SparseNanoBEIREvaluator(dataset_names=["msmarco", "nfcorpus", "nq"], batch_size=16)
# 7. Create a trainer & train
trainer = SparseEncoderTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
evaluator=dev_evaluator,
)
trainer.train()
# 8. Evaluate the model performance again after training
dev_evaluator(model)
# 9. Save the trained model
model.save_pretrained(f"models/{run_name}/final")
# 10. (Optional) Push it to the Hugging Face Hub
model.push_to_hub(run_name)
按此查看如何使用 Sentence Transformers,將模型微調為 reranker 模型
import logging
import traceback
import torch
from datasets import load_dataset
from sentence_transformers import SentenceTransformer
from sentence_transformers.cross_encoder import (
CrossEncoder,
CrossEncoderModelCardData,
CrossEncoderTrainer,
CrossEncoderTrainingArguments,
)
from sentence_transformers.cross_encoder.evaluation import (
CrossEncoderNanoBEIREvaluator,
CrossEncoderRerankingEvaluator,
)
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
from sentence_transformers.evaluation import SequentialEvaluator
from sentence_transformers.util import mine_hard_negatives
# Set the log level to INFO to get more information
logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
def main():
model_name = "jhu-clsp/mmBERT-small"
train_batch_size = 64
num_epochs = 1
num_hard_negatives = 5 # How many hard negatives should be mined for each question-answer pair
# 1a. Load a model to finetune with 1b. (Optional) model card data
model = CrossEncoder(
model_name,
model_card_data=CrossEncoderModelCardData(
language="en",
license="apache-2.0",
),
)
print("Model max length:", model.max_length)
print("Model num labels:", model.num_labels)
# 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq
logging.info("Read the gooaq training dataset")
full_dataset = load_dataset("sentence-transformers/gooaq", split="train").select(range(100_000))
dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"]
eval_dataset = dataset_dict["test"]
logging.info(train_dataset)
logging.info(eval_dataset)
# 2b. Modify our training dataset to include hard negatives using a very efficient embedding model
embedding_model = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1", device="cpu")
hard_train_dataset = mine_hard_negatives(
train_dataset,
embedding_model,
num_negatives=num_hard_negatives, # How many negatives per question-answer pair
margin=0, # Similarity between query and negative samples should be x lower than query-positive similarity
range_min=0, # Skip the x most similar samples
range_max=100, # Consider only the x most similar samples
sampling_strategy="top", # Sample the top negatives from the range
batch_size=4096, # Use a batch size of 4096 for the embedding model
output_format="labeled-pair", # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss
use_faiss=True,
)
logging.info(hard_train_dataset)
# 2c. (Optionally) Save the hard training dataset to disk
# hard_train_dataset.save_to_disk("gooaq-hard-train")
# Load again with:
# hard_train_dataset = load_from_disk("gooaq-hard-train")
# 3. Define our training loss.
# pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`
loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))
# 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking
nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(
dataset_names=["msmarco", "nfcorpus", "nq"],
batch_size=train_batch_size,
)
# 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs
# We include the positive answer in the list of negatives, so the evaluator can use the performance of the
# embedding model as a baseline.
hard_eval_dataset = mine_hard_negatives(
eval_dataset,
embedding_model,
corpus=full_dataset["answer"], # Use the full dataset as the corpus
num_negatives=30, # How many documents to rerank
batch_size=4096,
include_positives=True,
output_format="n-tuple",
use_faiss=True,
)
logging.info(hard_eval_dataset)
reranking_evaluator = CrossEncoderRerankingEvaluator(
samples=[
{
"query": sample["question"],
"positive": [sample["answer"]],
"documents": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],
}
for sample in hard_eval_dataset
],
batch_size=train_batch_size,
name="gooaq-dev",
# Realistic setting: only rerank the positives that the retriever found
# Set to True to rerank *all* positives
always_rerank_positives=False,
)
# 4c. Combine the evaluators & run the base model on them
evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])
evaluator(model)
# 5. Define the training arguments
short_model_name = model_name if "/" not in model_name else model_name.split("/")[-1]
run_name = f"reranker-{short_model_name}-gooaq-bce"
args = CrossEncoderTrainingArguments(
# Required parameter:
output_dir=f"models/{run_name}",
# Optional training parameters:
num_train_epochs=num_epochs,
per_device_train_batch_size=train_batch_size,
per_device_eval_batch_size=train_batch_size,
learning_rate=2e-5,
warmup_ratio=0.1,
fp16=False, # Set to False if you get an error that your GPU can't run on FP16
bf16=True, # Set to True if you have a GPU that supports BF16
dataloader_num_workers=4,
load_best_model_at_end=True,
metric_for_best_model="eval_gooaq-dev_ndcg@10",
# Optional tracking/debugging parameters:
eval_strategy="steps",
eval_steps=1000,
save_strategy="steps",
save_steps=1000,
save_total_limit=2,
logging_steps=200,
logging_first_step=True,
run_name=run_name, # Will be used in W&B if `wandb` is installed
seed=12,
)
# 6. Create the trainer & start training
trainer = CrossEncoderTrainer(
model=model,
args=args,
train_dataset=hard_train_dataset,
loss=loss,
evaluator=evaluator,
)
trainer.train()
# 7. Evaluate the final model, useful to include these in the model card
evaluator(model)
# 8. Save the final model
final_output_dir = f"models/{run_name}/final"
model.save_pretrained(final_output_dir)
# 9. (Optional) save the model to the Hugging Face Hub!
# It is recommended to run `huggingface-cli login` to log into your Hugging Face account first
try:
model.push_to_hub(run_name)
except Exception:
logging.error(
f"Error uploading model to the Hugging Face Hub:\n{traceback.format_exc()}To upload it manually, you can run "
f"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` "
f"and saving it using `model.push_to_hub('{run_name}')`."
)
if __name__ == "__main__":
main()
模型系列與連結
標準模型:
- mmBERT-small(總計 140M 個參數,42M 個非嵌入參數)
- mmBERT-base(總計 307M 個參數,110M 個非嵌入參數)
研究資源:
- 🤗 mmBERT 模型系列
- 📝 論文
- 🗂️ 訓練數據(3T+ 個 token,完全開放)
- 💻 GitHub 儲存庫
- 📊 訓練檢查點,可用於研究訓練或繼續預訓練
來源:Hugging Face Blog · huggingface.co