跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-10AI 評分41

UltraBERT:超越 BERT 預訓練的極限

UltraBERT: Beyond the Limit of BERT Pre-Training

AI 導讀

UltraBERT 以單一編碼器在分類與檢索任務均達最先進水平,GLUE 得分為 91.0,BEIR 為 47.7。模型在同一批次為低、高遮罩比例分設 LM head,並將知識蒸餾目標限制於低遮罩比例;研究指出,較低遮罩比例有利分類,較高比例則有利檢索。

正文

Published on Oct 10

Authors:

,

Abstract

Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance can be pushed by revisiting core design choices in training, revealing three key findings: (1) Encoders saturate on downstream tasks beyond a few trillion tokens, indicating that additional compute past this point does not help. (2) Across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval, contradicting prior work suggesting larger ratios for larger models. (3) Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones. Guided by these findings, we present UltraBERT, a single encoder that assigns separate LM heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. Together, these results show how far encoder performance can be pushed and provide a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.15028

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 3

dmis-lab/UltraBERT-150M

Fill-Mask •

0.2B •

Updated 26 minutes ago

•

80

•

6

dmis-lab/UltraBERT-380M

Fill-Mask •

0.4B •

Updated 26 minutes ago

•

101

•

5

dmis-lab/UltraBERT-Embed-150M

Sentence Similarity •

0.2B •

Updated 26 minutes ago

•

105

•

4

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.15028 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.15028 in a Space README.md to link it from this page.

Collections including this paper 1

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co