Ettin Suite 發佈 17M-1B 參數的配對編碼器與解碼器模型
Ettin Suite: SoTA Paired Encoders and Decoders
Ettin Suite 發佈涵蓋 17M-1B 參數的配對編碼器與解碼器模型,兩者使用相同的 2T tokens 公開數據及訓練配方。編碼器在各項任務及規模上超越 ModernBERT,解碼器則勝過或追平 Llama 3.2 和 SmolLM2。同條件測試中,編碼器在分類和檢索任務較佔優,解碼器在生成任務較佔優;模型涵蓋六種規模,訓練數據公開且可重現。
Ettin 在相同的數據、模型規模與訓練配方下比較編碼器和解碼器,呈現兩類架構在分類、檢索和生成任務上的不同優勢。
簡介
如果把 ModernBERT 的訓練配方套用到只含解碼器的模型,會有甚麼結果?答案是:一個最先進的解碼器語言模型,表現勝過 Llama 3.2 1B 和 SmolLM2!
我們推出一套全新的開放資料訓練配方,用來重現僅含編碼器的 ModernBERT 模型(而且實際表現更勝一籌!),再把完全相同的配方套用到僅含解碼器的模型。我們首次以同一配置訓練出兩款最先進模型,但採用兩種不同訓練目標:遮罩語言模型(MLM)和因果語言模型(CLM)。
這篇網誌介紹 Ettin,這是首個包含配對的僅編碼器與僅解碼器模型的 SoTA 模型系列(17M-1B 參數),以完全相同的資料(2T token)、架構和訓練配方訓練而成。Ettin 能真正公平地逐一比較不同架構,並讓兩類開放資料模型都達到最先進表現。我們亦進一步探討能否從解碼器出發,取得具競爭力的編碼器,反之亦然。
如果你想試用這些模型,網誌末段提供了一些程式範本!
編碼器與解碼器:架構之爭
大型語言模型社羣大致已聚焦於 GPT、Llama 和 Qwen 等僅含解碼器的模型。它們的生成能力令人印象深刻,但這種專注令其他類別較少受到關注,例如 BERT 等僅含編碼器的模型。
然而,類似 BERT 的編碼器模型仍是生產系統中分類、檢索和嵌入任務的主力。它們速度更快、記憶體效率更高,而且在判別式任務中通常更準確。關鍵差異在於注意力模式:
- 編碼器模型使用雙向注意力,讓序列中的每個 token 都能「看見」其他所有 token(完全可見)。
- 解碼器模型使用因果注意力;為了支援自回歸生成,每個 token 只能「看見」之前的 token。
解碼器模型迅速創新之際,編碼器模型的發展一度停滯;直到近期,ModernBERT 等研究工作為編碼器帶來現代化改良。但哪種架構較好?以往比較編碼器和解碼器時,所用資料集、架構及訓練配方各有不同,因此難以判斷。
Ettin 以北歐神話中的雙頭巨人命名,透過在相同資料、相同模型形狀和相同訓練配方上訓練兩種架構,實現受控比較。兩者只有注意力模式和訓練目標不同!
訓練配方:兩種架構均採用現代技術
我們以 ModernBERT 的訓練配方為基礎。該配方從僅含解碼器的模型借鑑現代技術,再引入編碼器訓練,為訓練這兩種架構提供穩固基礎。
模型規模
我們訓練了 6 種不同規模的模型,從 17M 到 1B 參數不等。這讓我們可以測試規模帶來的影響,也提供多款模型供你選用!無論你需要速度極快、可在裝置上運行的模型,還是功能強大但較慢的模型,我們都能滿足。
三階段訓練流程
我們採用完整的三階段訓練方法,盡量提升模型表現:
第一階段-預訓練(1.7T token):從多元且高品質的資料來源開始,以較短的上下文(1024 個 token)訓練,建立紮實的基礎知識。
第二階段-延長上下文(250B token):使用品質更高、經過篩選的資料,把上下文長度增至 8K token,讓模型理解較長文件和更複雜的關係。
第三階段-衰減(100B token):最後使用科學論文、教科書和精選內容等優質資料來源,同時逐步降低學習率。
現代架構組件
我們的編碼器模型具備 ModernBERT 的速度優勢,因此比上一代編碼器快得多。
資料來源與品質
與 ModernBERT 不同,我們所有訓練資料均公開且可重現:
你可以用新資料繼續訓練這些模型,或提出新的訓練配方進一步改善結果!
編碼器結果:勝過 ModernBERT
我們的編碼器模型在所有任務和模型規模上都勝過 ModernBERT,而且完全使用開放訓練資料。由於我們提供多種規模,你現在可以使用較小型的 ModernBERT 風格模型(適合在裝置上運行或快速推論),也可以選用性能強勁、足以壓倒對手的 1B 編碼器。
解碼器結果:勝過 Llama 3.2 和 SmolLM2
把同一配方套用到解碼器模型,也取得同樣出色的結果;我們的模型勝過或媲美既有基準,例如 Llama 3.2 和 SmolLM2:
在 SciQ 等知識密集型任務上,提升尤其明顯,反映出高品質訓練資料組合的好處。這些結果證明,我們的訓練配方能在兩種架構範式中都訓練出實力強勁的模型。
公平較量:在同等條件下比較編碼器與解碼器
我們首次能以完全相同的資料和配方,公平比較訓練所得的編碼器與解碼器架構。結果揭示了即使控制所有其他因素,仍然存在的根本架構優勢:
架構專屬優勢依然存在
結果呈現清晰趨勢:
編碼器在分類和檢索上佔優:在 MNLI 分類上,150M 編碼器(89.2)的表現甚至勝過 400M 解碼器(88.2)。在檢索任務上,差距較小但仍然明顯,尤其是解碼器未經 MNTP 訓練時。
解碼器擅長生成:在生成任務上,解碼器持續保持優勢,而且模型規模愈大,效能差距反而愈明顯。
規模並非總是關鍵:分類任務中,400M 編碼器勝過 1B 解碼器;生成任務中,400M 解碼器則勝過 1B 編碼器。
跨目標訓練表現不佳
由於缺乏新型編碼器模型,LLM2Vec 等研究提出使用 MLM 繼續預訓練解碼器。現在我們可以測試這種策略是否有效!
我們把訓練目標切換成相反目標,並額外訓練模型 50B token。結果如下:
- 由解碼器轉成編碼器:整體而言,分類/檢索表現仍落後於原生編碼器。
- 由編碼器轉成解碼器:表現明顯遜於原生解碼器,模型規模愈大時尤其如此。這可能是因為編碼器使用 MLM 而非 LLM2Vec 提出的 MNTP(遮罩下一個 token 預測)訓練;我們由解碼器轉成編碼器的配方也採用了 MNTP。
這表示架構選擇本身有根本影響,不只是訓練目標不同。
超越效能:瞭解模型行為
使用完全相同的訓練資料,我們可以研究不同目標如何影響學習。例如,透過 WinoGender 基準測試分析性別偏見,可見:
- 編碼器模型較常採用性別中立代名詞(中立代名詞超過 60%,解碼器則超過 30%)。
- 兩種架構都呈現男性偏見,但解碼器的偏見稍為明顯。
- 跨目標訓練會以可量度的方式影響偏見模式。
這為系統性研究訓練目標如何影響模型行為(不只影響準確率指標)開闢了新方向。
使用範例
只需幾行程式碼,就能使用這些模型!
編碼器
from transformers import AutoTokenizer, AutoModel
# Load encoder for classification/embeddings
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m")
def predict_masked_token(text):
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# Get predictions for [MASK] tokens
mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
predictions = outputs.logits[mask_indices]
# Get top 5 predictions
top_tokens = torch.topk(predictions, 5, dim=-1)
return [tokenizer.decode(token) for token in top_tokens.indices[0]]
# Example
masked_text = "The capital of France is [MASK]."
predictions = predict_masked_token(masked_text)
print(f"Predictions: {predictions}")
分類和檢索任務請使用編碼器模型:你亦可以使用專為這些任務微調的版本。
解碼器
文本生成任務請使用解碼器模型:
from transformers import AutoTokenizer, AutoModelForCausalLM
# Load decoder for generation
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")
# Generate text
prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(inputs.input_ids, max_length=50, temperature=0.7)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
微調範例
編碼器
按此查看如何使用 Sentence Transformers 把模型微調成密集嵌入模型
import argparse
from datasets import load_dataset
from sentence_transformers import (
SentenceTransformer,
SentenceTransformerTrainer,
SentenceTransformerTrainingArguments,
)
from sentence_transformers.evaluation import TripletEvaluator
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
from sentence_transformers.training_args import BatchSamplers
def main():
# parse the lr & model name
parser = argparse.ArgumentParser()
parser.add_argument("--lr", type=float, default=8e-5)
parser.add_argument("--model_name", type=str, default="jhu-clsp/ettin-encoder-150m")
args = parser.parse_args()
lr = args.lr
model_name = args.model_name
model_shortname = model_name.split("/")[-1]
# 1. Load a model to finetune
model = SentenceTransformer(model_name)
# 2. Load a dataset to finetune on
dataset = load_dataset(
"sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1",
"triplet-hard",
split="train",
)
dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"].select(range(1_250_000))
eval_dataset = dataset_dict["test"]
# 3. Define a loss function
loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16) # Increase mini_batch_size if you have enough VRAM
run_name = f"{model_shortname}-DPR-{lr}"
# 4. (Optional) Specify training arguments
args = SentenceTransformerTrainingArguments(
# Required parameter:
output_dir=f"output/{model_shortname}/{run_name}",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=512,
per_device_eval_batch_size=512,
warmup_ratio=0.05,
fp16=False, # Set to False if GPU can't handle FP16
bf16=True, # Set to True if GPU supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES, # (Cached)MultipleNegativesRankingLoss benefits from no duplicates
learning_rate=lr,
# Optional tracking/debugging parameters:
save_strategy="steps",
save_steps=500,
save_total_limit=2,
logging_steps=500,
run_name=run_name, # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed
)
# 5. (Optional) Create an evaluator & evaluate the base model
dev_evaluator = TripletEvaluator(
anchors=eval_dataset["query"],
positives=eval_dataset["positive"],
negatives=eval_dataset["negative"],
name="msmarco-co-condenser-dev",
)
dev_evaluator(model)
# 6. Create a trainer & train
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
evaluator=dev_evaluator,
)
trainer.train()
# 7. (Optional) Evaluate the trained model on the evaluator after training
dev_evaluator(model)
# 8. Save the model
model.save_pretrained(f"output/{model_shortname}/{run_name}/final")
# 9. (Optional) Push it to the Hugging Face Hub
model.push_to_hub(run_name, private=False)
if __name__ == "__main__":
main()
按此查看如何使用 PyLate 把模型微調成多向量嵌入模型
from datasets import load_dataset
from pylate import losses, models, utils
from sentence_transformers import (
SentenceTransformerTrainer,
SentenceTransformerTrainingArguments,
)
def main():
# Load the datasets required for knowledge distillation (train, queries, documents)
train = load_dataset(
path="lightonai/ms-marco-en-bge",
name="train",
)
queries = load_dataset(
path="lightonai/ms-marco-en-bge",
name="queries",
)
documents = load_dataset(
path="lightonai/ms-marco-en-bge",
name="documents",
)
# Set the transformation to load the documents/queries texts using the corresponding ids on the fly
train.set_transform(
utils.KDProcessing(queries=queries, documents=documents).transform,
)
# Define the base model, training parameters, and output directory
num_train_epochs = 1
lr = 8e-5
batch_size = 16
accum_steps = 1
model_name = "jhu-clsp/ettin-encoder-150m"
model_shortname = model_name.split("/")[-1]
# Set the run name for logging and output directory
run_name = f"{model_shortname}-colbert-KD-{lr}"
output_dir = f"output/{model_shortname}/{run_name}"
# Initialize the ColBERT model from the base model
model = models.ColBERT(model_name_or_path=model_name)
# Configure the training arguments (e.g., epochs, batch size, learning rate)
args = SentenceTransformerTrainingArguments(
output_dir=output_dir,
num_train_epochs=num_train_epochs,
per_device_train_batch_size=batch_size,
fp16=False, # Set to False if you get an error that your GPU can't run on FP16
bf16=True, # Set to True if you have a GPU that supports BF16
run_name=run_name,
logging_steps=10,
learning_rate=lr,
gradient_accumulation_steps=accum_steps,
warmup_ratio=0.05,
)
# Use the Distillation loss function for training
train_loss = losses.Distillation(model=model)
# Initialize the trainer
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train,
loss=train_loss,
data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),
)
# Start the training process
trainer.train()
model.save_pretrained(f"{output_dir}/final")
if __name__ == "__main__":
main()
按此查看如何使用 Sentence Transformers 把模型微調成稀疏檢索模型
import logging
from datasets import load_dataset
from sentence_transformers import (
SparseEncoder,
SparseEncoderModelCardData,
SparseEncoderTrainer,
SparseEncoderTrainingArguments,
)
from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
from sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss
from sentence_transformers.training_args import BatchSamplers
logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
# 1. Load a model to finetune with 2. (Optional) model card data
model = SparseEncoder(
"jhu-clsp/ettin-encoder-150m",
model_card_data=SparseEncoderModelCardData(
language="en",
license="apache-2.0",
)
)
# 3. Load a dataset to finetune on
full_dataset = load_dataset("sentence-transformers/natural-questions", split="train").select(range(100_000))
dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"]
eval_dataset = dataset_dict["test"]
# 4. Define a loss function
loss = SpladeLoss(
model=model,
loss=SparseMultipleNegativesRankingLoss(model=model),
query_regularizer_weight=5e-5,
document_regularizer_weight=3e-5,
)
# 5. (Optional) Specify training arguments
run_name = "splade-distilbert-base-uncased-nq"
args = SparseEncoderTrainingArguments(
# Required parameter:
output_dir=f"models/{run_name}",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
learning_rate=2e-5,
warmup_ratio=0.1,
fp16=True, # Set to False if you get an error that your GPU can't run on FP16
bf16=False, # Set to True if you have a GPU that supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES, # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
# Optional tracking/debugging parameters:
eval_strategy="steps",
eval_steps=1000,
save_strategy="steps",
save_steps=1000,
save_total_limit=2,
logging_steps=200,
run_name=run_name, # Will be used in W&B if `wandb` is installed
)
# 6. (Optional) Create an evaluator & evaluate the base model
dev_evaluator = SparseNanoBEIREvaluator(dataset_names=["msmarco", "nfcorpus", "nq"], batch_size=16)
# 7. Create a trainer & train
trainer = SparseEncoderTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
evaluator=dev_evaluator,
)
trainer.train()
# 8. Evaluate the model performance again after training
dev_evaluator(model)
# 9. Save the trained model
model.save_pretrained(f"models/{run_name}/final")
# 10. (Optional) Push it to the Hugging Face Hub
model.push_to_hub(run_name)
按此查看如何使用 Sentence Transformers 把模型微調成重排序器模型
import logging
import traceback
import torch
from datasets import load_dataset
from sentence_transformers import SentenceTransformer
from sentence_transformers.cross_encoder import (
CrossEncoder,
CrossEncoderModelCardData,
CrossEncoderTrainer,
CrossEncoderTrainingArguments,
)
from sentence_transformers.cross_encoder.evaluation import (
CrossEncoderNanoBEIREvaluator,
CrossEncoderRerankingEvaluator,
)
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
from sentence_transformers.evaluation import SequentialEvaluator
from sentence_transformers.util import mine_hard_negatives
# Set the log level to INFO to get more information
logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
def main():
model_name = "jhu-clsp/ettin-encoder-150m"
train_batch_size = 64
num_epochs = 1
num_hard_negatives = 5 # How many hard negatives should be mined for each question-answer pair
# 1a. Load a model to finetune with 1b. (Optional) model card data
model = CrossEncoder(
model_name,
model_card_data=CrossEncoderModelCardData(
language="en",
license="apache-2.0",
),
)
print("Model max length:", model.max_length)
print("Model num labels:", model.num_labels)
# 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq
logging.info("Read the gooaq training dataset")
full_dataset = load_dataset("sentence-transformers/gooaq", split="train").select(range(100_000))
dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
train_dataset = dataset_dict["train"]
eval_dataset = dataset_dict["test"]
logging.info(train_dataset)
logging.info(eval_dataset)
# 2b. Modify our training dataset to include hard negatives using a very efficient embedding model
embedding_model = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1", device="cpu")
hard_train_dataset = mine_hard_negatives(
train_dataset,
embedding_model,
num_negatives=num_hard_negatives, # How many negatives per question-answer pair
margin=0, # Similarity between query and negative samples should be x lower than query-positive similarity
range_min=0, # Skip the x most similar samples
range_max=100, # Consider only the x most similar samples
sampling_strategy="top", # Sample the top negatives from the range
batch_size=4096, # Use a batch size of 4096 for the embedding model
output_format="labeled-pair", # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss
use_faiss=True,
)
logging.info(hard_train_dataset)
# 2c. (Optionally) Save the hard training dataset to disk
# hard_train_dataset.save_to_disk("gooaq-hard-train")
# Load again with:
# hard_train_dataset = load_from_disk("gooaq-hard-train")
# 3. Define our training loss.
# pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`
loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))
# 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking
nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(
dataset_names=["msmarco", "nfcorpus", "nq"],
batch_size=train_batch_size,
)
# 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs
# We include the positive answer in the list of negatives, so the evaluator can use the performance of the
# embedding model as a baseline.
hard_eval_dataset = mine_hard_negatives(
eval_dataset,
embedding_model,
corpus=full_dataset["answer"], # Use the full dataset as the corpus
num_negatives=30, # How many documents to rerank
batch_size=4096,
include_positives=True,
output_format="n-tuple",
use_faiss=True,
)
logging.info(hard_eval_dataset)
reranking_evaluator = CrossEncoderRerankingEvaluator(
samples=[
{
"query": sample["question"],
"positive": [sample["answer"]],
"documents": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],
}
for sample in hard_eval_dataset
],
batch_size=train_batch_size,
name="gooaq-dev",
# Realistic setting: only rerank the positives that the retriever found
# Set to True to rerank *all* positives
always_rerank_positives=False,
)
# 4c. Combine the evaluators & run the base model on them
evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])
evaluator(model)
# 5. Define the training arguments
short_model_name = model_name if "/" not in model_name else model_name.split("/")[-1]
run_name = f"reranker-{short_model_name}-gooaq-bce"
args = CrossEncoderTrainingArguments(
# Required parameter:
output_dir=f"models/{run_name}",
# Optional training parameters:
num_train_epochs=num_epochs,
per_device_train_batch_size=train_batch_size,
per_device_eval_batch_size=train_batch_size,
learning_rate=2e-5,
warmup_ratio=0.1,
fp16=False, # Set to False if you get an error that your GPU can't run on FP16
bf16=True, # Set to True if you have a GPU that supports BF16
dataloader_num_workers=4,
load_best_model_at_end=True,
metric_for_best_model="eval_gooaq-dev_ndcg@10",
# Optional tracking/debugging parameters:
eval_strategy="steps",
eval_steps=1000,
save_strategy="steps",
save_steps=1000,
save_total_limit=2,
logging_steps=200,
logging_first_step=True,
run_name=run_name, # Will be used in W&B if `wandb` is installed
seed=12,
)
# 6. Create the trainer & start training
trainer = CrossEncoderTrainer(
model=model,
args=args,
train_dataset=hard_train_dataset,
loss=loss,
evaluator=evaluator,
)
trainer.train()
# 7. Evaluate the final model, useful to include these in the model card
evaluator(model)
# 8. Save the final model
final_output_dir = f"models/{run_name}/final"
model.save_pretrained(final_output_dir)
# 9. (Optional) save the model to the Hugging Face Hub!
# It is recommended to run `huggingface-cli login` to log into your Hugging Face account first
try:
model.push_to_hub(run_name)
except Exception:
logging.error(
f"Error uploading model to the Hugging Face Hub:\n{traceback.format_exc()}To upload it manually, you can run "
f"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` "
f"and saving it using `model.push_to_hub('{run_name}')`."
)
if __name__ == "__main__":
main()
解碼器
按此展開解碼器訓練程式碼
完整訓練
python trl/scripts/sft.py \
--model_name_or_path jhu-clsp/ettin-decoder-17m \
--dataset_name trl-lib/Capybara \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--packing \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing \
--eos_token '<|im_end|>' \
--eval_strategy steps \
--eval_steps 100 \
--output_dir ettin-decoder-17m \
--push_to_hub
LoRA
python trl/scripts/sft.py \
--model_name_or_path jhu-clsp/ettin-decoder-17m \
--dataset_name trl-lib/Capybara \
--learning_rate 2.0e-4 \
--num_train_epochs 1 \
--packing \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing \
--eos_token '<|im_end|>' \
--eval_strategy steps \
--eval_steps 100 \
--use_peft \
--lora_r 32 \
--lora_alpha 16 \
--output_dir ettin-decoder-17m \
--push_to_hub
使用 sft.py:
import argparse
from datasets import load_dataset
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from transformers.models.auto.modeling_auto import MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES
from trl import (
ModelConfig,
ScriptArguments,
SFTConfig,
SFTTrainer,
TrlParser,
clone_chat_template,
get_kbit_device_map,
get_peft_config,
get_quantization_config,
)
def main(script_args, training_args, model_args):
################
# Model init kwargs & Tokenizer
################
quantization_config = get_quantization_config(model_args)
model_kwargs = dict(
revision=model_args.model_revision,
trust_remote_code=model_args.trust_remote_code,
attn_implementation=model_args.attn_implementation,
torch_dtype=model_args.torch_dtype,
use_cache=False if training_args.gradient_checkpointing else True,
device_map=get_kbit_device_map() if quantization_config is not None else None,
quantization_config=quantization_config,
)
# Create model
config = AutoConfig.from_pretrained(model_args.model_name_or_path)
valid_image_text_architectures = MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.values()
if config.architectures and any(arch in valid_image_text_architectures for arch in config.architectures):
from transformers import AutoModelForImageTextToText
model_kwargs.pop("use_cache", None) # Image models do not support cache
model = AutoModelForImageTextToText.from_pretrained(model_args.model_name_or_path, **model_kwargs)
else:
model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, **model_kwargs)
# Create tokenizer
tokenizer = AutoTokenizer.from_pretrained(
model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, use_fast=True
)
# Set default chat template if needed
if tokenizer.chat_template is None:
# TODO: source should be passed as an argument
model, tokenizer = clone_chat_template(model, tokenizer, "Qwen/Qwen3-0.6B")
################
# Dataset
################
dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)
################
# Training
################
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset[script_args.dataset_train_split],
eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != "no" else None,
processing_class=tokenizer,
peft_config=get_peft_config(model_args),
)
trainer.train()
# Save and push to hub
trainer.save_model(training_args.output_dir)
if training_args.push_to_hub:
trainer.push_to_hub(dataset_name=script_args.dataset_name)
def make_parser(subparsers: argparse._SubParsersAction = None):
dataclass_types = (ScriptArguments, SFTConfig, ModelConfig)
if subparsers is not None:
parser = subparsers.add_parser("sft", help="Run the SFT training script", dataclass_types=dataclass_types)
else:
parser = TrlParser(dataclass_types)
return parser
if __name__ == "__main__":
parser = make_parser()
# When using the trl cli, this script may be run with additional arguments, corresponding accelerate arguments.
# To ensure that their parsing does not interfere with the script arguments, parse the arguments with
# `return_remaining_strings=True`, then ignore the remaining strings.
script_args, training_args, model_args, _ = parser.parse_args_and_config(return_remaining_strings=True)
main(script_args, training_args, model_args)
模型系列與連結
完整 Ettin 系列包含 6 種不同規模的模型(編碼器和解碼器各有一款):
標準模型:
- ettin-encoder-17m / ettin-decoder-17m(17M 參數)
- ettin-encoder-32m / ettin-decoder-32m(32M 參數)
- ettin-encoder-68m / ettin-decoder-68m(68M 參數)
- ettin-encoder-150m / ettin-decoder-150m(150M 參數)
- ettin-encoder-400m / ettin-decoder-400m(400M 參數)
- ettin-encoder-1b / ettin-decoder-1b(1B 參數)
研究資源:
- 🤗 Ettin 模型集合
- 📝 論文
- 🗂️ 訓練資料(2T+ token,完全開放)
- 💻 GitHub 程式碼庫
- 📊 250+ 個訓練檢查點,可用於研究訓練動態或知識學習
來源:Hugging Face Blog · huggingface.co





