Google 發佈 EmbeddingGemma 多語言嵌入模型
Welcome EmbeddingGemma, Google's new efficient embedding model
Google 發佈 EmbeddingGemma,一款支援 100 多種語言、具備 308M parameters 和 2K context window 的多語言嵌入模型。文章示範以 Sentence Transformers、LangChain 等工具接入模型,並在 MIRIAD 醫療檢索測試中將微調版的 NDCG@10 提升至 0.8862,基礎模型為 0.8340。
文章提供 EmbeddingGemma 接入多個檢索框架的程式示例,並説明查詢與文件提示詞的設定方式,方便將模型納入既有檢索流程。
重點摘要
Google 今日發布EmbeddingGemma,這是一款最先進的多語言嵌入模型,非常適合裝置端使用情境。模型專為速度和效率而設計,體積精巧,只有308M 個參數和2K 上下文窗口,為流動 RAG 流程、代理等開拓新可能。EmbeddingGemma 經訓練支援超過100 種語言;撰寫本文時,它是 Massive Text Embedding Benchmark(MTEB)上 500M 以下排名最高的純文字多語言嵌入模型。
目錄
簡介
文本嵌入已成為現代自然語言應用的基礎,能將詞語、句子和文件轉換成密集向量,捕捉其含義、情感和意圖。這些向量可在大型語料庫中快速進行相似度搜尋、羣集分析、分類和檢索,支援推薦引擎、語義搜尋、檢索增強生成和程式碼搜尋工具等各種應用。用來計算這些嵌入的模型廣泛使用,Hugging Face 每月下載量已遠超 200 million 次。
Google DeepMind 的EmbeddingGemma建基於這項基礎,成為最新、能力最強的小型多語言嵌入模型。它只有 308M 個參數、支援 2k token 上下文窗口和超過 100 種語言,在 Massive Multilingual Text Embedding Benchmark(MMTEB)上達到最先進表現,量化後記憶體用量仍低於 200 MB。
這些設計選擇共同打造出實用的開源工具,可在日常裝置上計算高品質多語言嵌入。
本文會介紹 EmbeddingGemma 的架構和訓練方式,並示範如何使用 Sentence Transformers、LangChain、LlamaIndex、Haystack、txtai、Transformers.js、Text Embedding Inference 和 ONNX 等多種框架操作模型。
接着,我們會示範如何在你的領域微調 EmbeddingGemma,以進一步提升表現。範例會使用 Medical Instruction and Retrieval Dataset(MIRIAD)微調 EmbeddingGemma。所得模型 sentence-transformers/embeddinggemma-300m-medical 在我們的任務中達到最先進表現:根據詳細醫療問題,檢索科學醫學論文中的相關段落。在這項任務中,它甚至勝過大一倍的模型。
架構
EmbeddingGemma 建基於 Gemma3 transformers 主幹,但將因果(單向)注意力改為雙向注意力。這代表序列中的前置 token 可以關注後續 token,令架構由 decoder 實際轉變為 encoder。在檢索等嵌入任務中,Encoder 模型可勝過作為 decoder 的 LLM(Weller 等人,2025)。採用此主幹後,模型一次可處理多達 2048 個 token,足以應付常見檢索輸入,尤其是較長輸入往往會令文本嵌入資訊流失。
除 Gemma3 新式 encoder 主幹會產生 token 嵌入外,模型亦使用平均池化層將 token 嵌入轉成文本嵌入。最後,兩個密集層會將文本嵌入轉換成 768 維向量。
EmbeddingGemma 使用 Matryoshka 表徵學習(MRL)訓練,因此可按需要將 768 維輸出截短至 512、256 或 128 維。這能加快下游處理並減少記憶體和磁碟空間用量。請參閲Sentence Transformers 用法,其中有示範截短方式的程式碼片段。
模型以經仔細整理、總計約 320 billion 個 token 的多語言語料庫訓練。專有資料集結合公開網絡文本、程式碼、技術文件和特定任務合成範例,並經過篩選,以排除兒童性虐待材料(CSAM)、敏感資料,以及低品質或不安全內容。
評估
EmbeddingGemma 以涵蓋廣泛任務、領域和語言的 MMTEB(Multilingual, v2)和 MTEB(English, v2)測試套件作基準評估。儘管模型規模只有 308M 參數,它仍持續勝過可比基準模型,同時保持極小的記憶體佔用。
| MTEB(Multilingual, v2)表現 | MTEB(English, v2)表現 |
|---|---|
|
|
結果將列於官方MTEB 排行榜。為減少潛在過度擬合,我們排除任何使用超過 20% MTEB 資料訓練的模型。
示範
示範亦可用全螢幕體驗。
你的瀏覽器不支援 video 標籤。
歡迎在桌面裝置上親自體驗示範。
用法
EmbeddingGemma 已整合至許多熱門工具,容易融入現有工作流程和應用程式。模型已整合至 Sentence Transformers,因此也支援 LangChain、LlamaIndex、Haystack 和 txtai 等以 Sentence Transformers 為底層的專案。以下範例可讓你開始使用偏好的框架。
若要用於生產環境,可透過 Text Embeddings Inference(TEI)在多種硬件設定上高效率提供模型服務;網頁應用程式則可使用 Transformers.js。
無論選用哪種框架,都應留意提示。嵌入模型會將提示加在輸入文本前,讓模型區分不同任務。EmbeddingGemma 以這些提示名稱和提示內容訓練,因此使用模型時亦應包含:
query:"task: search result | query: ",document:"title: none | text: ",BitextMining:"task: search result | query: ",Clustering:"task: clustering | query: ",Classification:"task: classification | query: ",InstructionRetrieval:"task: code retrieval | query: ",MultilabelClassification:"task: classification | query: ",PairClassification:"task: sentence similarity | query: ",Reranking:"task: search result | query: ",Retrieval-query:"task: search result | query: ",Retrieval-document:"title: none | text: ",STS:"task: sentence similarity | query: ",Summarization:"task: summarization | query: "
在 Sentence Transformers 中,query 和 document 提示會在呼叫 model.encode_query 和 model.encode_document 時自動使用;但在其他框架中,你可能需要:
- 指定提示名稱(例如「Reranking」),
- 指定提示字串(例如「task: search result | query: 」),或
- 手動將提示加在輸入文本前。
以下範例腳本會示範如何搭配多種框架使用。
Sentence Transformers
請安裝以下套件:
pip install git+https://github.com/huggingface/transformers@v4.56.0-Embedding-Gemma-preview
pip install sentence-transformers>=5.0.0
檢索
使用 Sentence Transformers 進行推論相當簡單,語義搜尋範例如下:
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("google/embeddinggemma-300m")
# Run inference with queries and documents
query = "Which planet is known as the Red Planet?"
documents = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (768,) (4, 768)
# Compute similarities to determine a ranking
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.3011, 0.6359, 0.4930, 0.4889]])
# Convert similarities to a ranking
ranking = similarities.argsort(descending=True)[0]
print(ranking)
# tensor([1, 2, 3, 0])
- Sentence Transformers
encode_query方法文件 - Sentence Transformers
encode_document方法文件 - Sentence Transformers
similarity方法文件
點擊查看非檢索程式碼
如果你不打算將此模型用於資訊檢索,最適合的做法可能是使用最通用的 encode 方法,並搭配最能描述下游任務的模型提示,選項如下:
BitextMining:找出兩種語言中的翻譯句子配對。Clustering:找出相似文本並將它們分組。Classification:為文本指派預先定義的標籤。InstructionRetrieval:根據自然語言指令檢索相關程式碼片段。MultilabelClassification:為文本指派多個標籤。PairClassification:為文本指派預先定義的標籤。Reranking:根據相關性重新排序搜尋結果。Retrieval-query:根據查詢檢索文件。Retrieval-document:根據文件內容檢索文件。STS:計算文本之間的語義相似度。Summarization:生成精簡摘要。
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("google/embeddinggemma-300m")
# Let's inspect the configured prompts
print(model.prompts)
# {
# "query": "task: search result | query: ",
# "document": "title: none | text: ",
# "BitextMining": "task: search result | query: ",
# "Clustering": "task: clustering | query: ",
# "Classification": "task: classification | query: ",
# "InstructionRetrieval": "task: code retrieval | query: ",
# "MultilabelClassification": "task: classification | query: ",
# "PairClassification": "task: sentence similarity | query: ",
# "Reranking": "task: search result | query: ",
# "Retrieval-query": "task: search result | query: ",
# "Retrieval-document": "title: none | text: ",
# "STS": "task: sentence similarity | query: ",
# "Summarization": "task: summarization | query: ",
# }
# Compute semantic textual similarity using texts, so let's use the STS prompt
texts = [
"The weather is beautiful today.",
"It's a lovely day outside.",
"The stock market crashed yesterday.",
"I enjoy programming with Python."
]
embeddings = model.encode(texts, prompt_name="STS")
print(embeddings.shape)
# (4, 768)
# Compute similarities
similarities = model.similarity(embeddings, embeddings)
print(similarities)
"""
tensor([[1.0000, 0.9305, 0.4660, 0.4326],
[0.9305, 1.0000, 0.4227, 0.4434],
[0.4660, 0.4227, 1.0000, 0.2638],
[0.4326, 0.4434, 0.2638, 1.0000]])
"""
點擊查看如何截短嵌入維度,以加快搜尋並降低成本
由於 google/embeddinggemma-300m 使用 MRL 訓練,截短此模型生成的嵌入向量至較低維度時,評估表現不會明顯下降。較低維度的嵌入向量佔用較少磁碟和記憶體空間,檢索、羣集分析或分類等下游任務也會更快。
在 Sentence Transformers 中,可透過 truncate_dim 參數設定較低維度;此參數可在初始化 SentenceTransformer 時設定,亦可在呼叫 model.encode/model.encode_query/model.encode_document 時設定:
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("google/embeddinggemma-300m", truncate_dim=256)
# Run inference with queries and documents
query = "Which planet is known as the Red Planet?"
documents = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (256,) (4, 256)
# Compute similarities to determine a ranking
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4016, 0.6715, 0.5283, 0.5261]])
# Convert similarities to a ranking
ranking = similarities.argsort(descending=True)[0]
print(ranking)
# tensor([1, 2, 3, 0])
請注意,即使嵌入向量比完整尺寸小 3 倍,排序仍會維持不變。
LangChain
你亦可選用 LangChain 的 HuggingFaceEmbeddings,其底層使用 Sentence Transformers。請注意,你需要告訴 LangChain,查詢和文件應分別使用名為「query」和「document」的提示。此範例展示簡單資訊檢索設定,但同一嵌入模型亦可用於更複雜的情境。
請安裝以下套件:
pip install git+https://github.com/huggingface/transformers@v4.56.0-Embedding-Gemma-preview
pip install sentence-transformers
pip install langchain
pip install langchain-community
pip install langchain-huggingface
pip install faiss-cpu
from langchain.docstore.document import Document
from langchain_community.vectorstores import FAISS
from langchain_huggingface.embeddings import HuggingFaceEmbeddings
# Download the model from the 🤗 Hub. Also specify to use the "query" and "document" prompts
# as defined in the model configuration, as LangChain doesn't automatically use them.
# See https://huggingface.co/google/embeddinggemma-300m/blob/main/config_sentence_transformers.json
embedder = HuggingFaceEmbeddings(
model_name="google/embeddinggemma-300m",
query_encode_kwargs={"prompt_name": "query"},
encode_kwargs={"prompt_name": "document"}
)
data = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
# Create documents for the vector store
documents = [Document(page_content=text, metadata={"id": i}) for i, text in enumerate(data)]
# Create vector store using FAISS. Setting distance_strategy to "MAX_INNER_PRODUCT" uses
# FAISS' FlatIndexIP behind the scenes, which is optimized for inner product search. This
# is what the model was trained for
vector_store = FAISS.from_documents(documents, embedder, distance_strategy="MAX_INNER_PRODUCT")
# Search for top 3 similar documents
query = "Which planet is known as the Red Planet?"
results = vector_store.similarity_search_with_score(query, k=3)
# Print results
for doc, score in results:
print(f"Text: {doc.page_content} (score: {score:.4f})")
"""
Text: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)
Text: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)
Text: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)
"""
LlamaIndex
EmbeddingGemma 亦支援 LlamaIndex,因為它底層使用 Sentence Transformers。為確保正確運作,必須按照模型設定指定查詢和文件提示,否則表現會不理想。此腳本示範以 LlamaIndex 使用 EmbeddingGemma 的基本方式;較複雜的情境亦可使用 HuggingFaceEmbedding 類別。
請安裝以下套件:
pip install git+https://github.com/huggingface/transformers@v4.56.0-Embedding-Gemma-preview
pip install sentence-transformers
pip install llama-index
pip install llama-index-embeddings-huggingface
pip install llama-index-vector-stores-faiss
import faiss
from llama_index.core.schema import TextNode
from llama_index.core.vector_stores import VectorStoreQuery
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.vector_stores.faiss import FaissVectorStore
# Download from the 🤗 Hub. Also specify the query and document prompts as
# defined in the model configuration, as LlamaIndex doesn't automatically load them.
# See https://huggingface.co/google/embeddinggemma-300m/blob/main/config_sentence_transformers.json
embeddings = HuggingFaceEmbedding(
model_name="google/embeddinggemma-300m",
query_instruction="task: search result | query: ",
text_instruction="title: none | text: ",
)
data = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
# Create a sample vector store
store = FaissVectorStore(faiss_index=faiss.IndexFlatIP(768))
store.add([TextNode(id=i, text=text, embedding=embeddings.get_text_embedding(text)) for i, text in enumerate(data)])
# Search for top k similar documents
query = "Which planet is known as the Red Planet?"
query_embedding = embeddings.get_query_embedding(query)
results = store.query(VectorStoreQuery(query_embedding=query_embedding, similarity_top_k=3))
# Print results
for idx, score in zip(results.ids, results.similarities):
print(f"Text: {data[int(idx)]} (score: {score:.4f})")
"""
Text: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)
Text: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)
Text: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)
"""
Haystack
EmbeddingGemma 亦可搭配 Haystack 使用。Haystack 是用來建立可供生產環境使用的搜尋及語言應用程式框架。與 LangChain 和 LlamaIndex 一樣,Haystack 底層使用 Sentence Transformers,需要指定合適提示才能讓 EmbeddingGemma 發揮最佳效能。以下範例示範如何使用 EmbeddingGemma 和 Haystack 建立基本檢索流程。
請安裝以下套件:
pip install git+https://github.com/huggingface/transformers@v4.56.0-Embedding-Gemma-preview
pip install sentence-transformers
pip install haystack-ai
from haystack import Document, Pipeline
from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder
from haystack.components.retrievers import InMemoryEmbeddingRetriever
from haystack.document_stores.in_memory import InMemoryDocumentStore
# Initialize the document store
document_store = InMemoryDocumentStore()
# Initialize the document and query embedders
document_embedder = SentenceTransformersDocumentEmbedder(
model="google/embeddinggemma-300m", encode_kwargs={"prompt_name": "document"}
)
query_embedder = SentenceTransformersTextEmbedder(
model="google/embeddinggemma-300m", encode_kwargs={"prompt_name": "query"}
)
document_embedder.warm_up()
query_embedder.warm_up()
data = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
]
# Convert to Haystack documents and write to document store
documents = [Document(content=text, id=str(i)) for i, text in enumerate(data)]
documents_with_embeddings = document_embedder.run(documents=documents)["documents"]
document_store.write_documents(documents_with_embeddings)
# Create a query pipeline using a query embedder and compatible retriever
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", query_embedder)
query_pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store=document_store, top_k=3))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
# Search for top 3 similar documents
query = "Which planet is known as the Red Planet?"
results = query_pipeline.run({"text_embedder": {"text": query}})
# Print results
for document in results["retriever"]["documents"]:
print(f"Text: {document.content} (score: {document.score:.4f})")
"""
Text: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)
Text: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)
Text: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)
"""
txtai
txtai 亦相容於 EmbeddingGemma。與其他框架一樣,txtai 底層使用 Sentence Transformers;為令 EmbeddingGemma 發揮最佳效能,需要指定合適提示。以下範例示範如何使用 txtai 建立基本檢索系統。
請安裝以下套件:
pip install git+https://github.com/huggingface/transformers@v4.56.0-Embedding-Gemma-preview
pip install sentence-transformers
pip install txtai
from txtai import Embeddings
# Download from the 🤗 Hub. Also specify the query and document prompts as
# defined in the model configuration, as txtai doesn't automatically load them.
# See https://huggingface.co/google/embeddinggemma-300m/blob/main/config_sentence_transformers.json
embeddings = Embeddings(
path="google/embeddinggemma-300m",
method="sentence-transformers",
instructions={
"query": "task: search result | query: ",
"data": "title: none | text: ",
}
)
data = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
# Create a sample vector store
embeddings.index(data)
# Search for top k similar documents
query = "Which planet is known as the Red Planet?"
results = embeddings.search(query, 3)
# Print results
for idx, score in results:
print(f"Text: {data[int(idx)]} (score: {score:.4f})")
"""
Text: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)
Text: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)
Text: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)
"""
Transformers.js
你甚至可透過 Transformers.js,在瀏覽器中以 100% 本機方式執行 EmbeddingGemma!如果尚未安裝,可使用以下方式從 NPM 安裝程式庫:
npm i @huggingface/transformers
接着可以按以下方式計算嵌入:
import { AutoModel, AutoTokenizer, matmul } from "@huggingface/transformers";
// Download from the 🤗 Hub
const model_id = "onnx-community/embeddinggemma-300m-ONNX";
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const model = await AutoModel.from_pretrained(model_id, {
dtype: "fp32", // Options: "fp32" | "q8" | "q4"
});
// Run inference with queries and documents
const prefixes = {
query: "task: search result | query: ",
document: "title: none | text: ",
};
const query = prefixes.query + "Which planet is known as the Red Planet?";
const documents = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
].map((x) => prefixes.document + x);
const inputs = await tokenizer([query, ...documents], { padding: true });
const { sentence_embedding } = await model(inputs);
// Compute similarities to determine a ranking
const scores = await matmul(sentence_embedding, sentence_embedding.transpose(1, 0));
const similarities = scores.tolist()[0].slice(1);
console.log(similarities);
// [ 0.30109718441963196, 0.6358831524848938, 0.4930494725704193, 0.48887503147125244 ]
// Convert similarities to a ranking
const ranking = similarities.map((score, index) => ({ index, score })).sort((a, b) => b.score - a.score);
console.log(ranking);
// [
// { index: 1, score: 0.6358831524848938 },
// { index: 2, score: 0.4930494725704193 },
// { index: 3, score: 0.48887503147125244 },
// { index: 0, score: 0.30109718441963196 }
// ]
Text Embeddings Inference
你可使用 Text Embeddings Inference(TEI)版本 1.8.1 或更新版本,輕鬆將 EmbeddingGemma 部署至開發和生產環境。
- CPU:
docker run -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.1 --model-id google/embeddinggemma-300m --dtype float32
- CPU(搭配 ONNX Runtime):
docker run -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.1 --model-id onnx-community/embeddinggemma-300m-ONNX --dtype float32 --pooling mean
- NVIDIA CUDA:
docker run --gpus all --shm-size 1g -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:cuda-1.8.1 --model-id google/embeddinggemma-300m --dtype float32
如果使用帶有
cuda-1.8.1標籤的 Docker 容器,便支援多種 GPU 架構:Turing、Ampere、Ada Lovelace 和 Hopper。若想使用專為你的 GPU 設計、體積更小的映像,也可選擇特定標籤,例如turing-1.8.1、1.8.1和86-1.8.1(Ampere)、89-1.8.1(Ada Lovelace),或hopper-1.8.1。
部署完成後,無論使用哪種裝置或執行環境,都可透過 /v1/embeddings 端點,按照 OpenAI Embeddings API 規格生成嵌入。
curl http://0.0.0.0:8080/v1/embeddings -H "Content-Type: application/json" -d '{"model":"google/embeddinggemma-300m","input":["task: search result | query: Which planet is known as the Red Planet?","task: search result | query: Where did Amelia Earhart first fly?"]}'
你亦可使用 /embed 端點,這是 Text Embeddings Inference Embeddings API 的一部分。它支援 prompt_name 參數,因此無須手動將提示加在輸入前,只需透過 prompt_name 選取提示即可。
curl http://0.0.0.0:8080/embed -H "Content-Type: application/json" -d '{"inputs":["Which planet is known as the Red Planet?","Where did Amelia Earthart first fly?"],"prompt_name":"query","normalize":true}'
此外,請注意,由於
google/embeddinggemma-300m使用 Matryoshka 表徵學習(MRL)訓練,你亦可使用dimensions參數,透過/v1/embeddings和/embed端點將嵌入截短至較低維度(512、256 和 128),而不影響評估表現。
ONNX Runtime
你亦可直接使用 ONNX Runtime 執行模型,令模型具備高度可攜性並支援跨平台。以下示範使用 Python,但同樣方法亦適用於其他語言(Java、C#、C++ 等)。
from huggingface_hub import hf_hub_download
import onnxruntime as ort
from transformers import AutoTokenizer
# Download from the 🤗 Hub
model_id = "onnx-community/embeddinggemma-300m-ONNX"
model_path = hf_hub_download(model_id, subfolder="onnx", filename="model.onnx") # Download graph
hf_hub_download(model_id, subfolder="onnx", filename="model.onnx_data") # Download weights
session = ort.InferenceSession(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Run inference with queries and documents
prefixes = {
"query": "task: search result | query: ",
"document": "title: none | text: ",
}
query = prefixes["query"] + "Which planet is known as the Red Planet?"
documents = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
documents = [prefixes["document"] + x for x in documents]
inputs = tokenizer([query] + documents, padding=True, return_tensors="np")
_, sentence_embedding = session.run(None, inputs.data)
print(sentence_embedding.shape) # (5, 768)
# Compute similarities to determine a ranking
query_embeddings = sentence_embedding[0]
document_embeddings = sentence_embedding[1:]
similarities = query_embeddings @ document_embeddings.T
print(similarities) # [0.30109745 0.635883 0.49304956 0.48887485]
# Convert similarities to a ranking
ranking = similarities.argsort()[::-1]
print(ranking) # [1 2 3 0]
微調
與所有相容 Sentence Transformers 程式庫的模型一樣,EmbeddingGemma 可輕鬆在特定資料集上微調。為作示範,我們會微調 google/embeddinggemma-300m,使用 Medical Instruction and Retrieval Dataset(MIRIAD)資料集,令微調後模型特別擅長根據詳細醫療問題,從科學醫學論文中找出最多 1000 個 token 的段落。這些段落可作為生成模型的重要上下文,協助它更有效回答問題。
以下可透過可展開分頁逐一探索微調流程中的各個重要組件。每個分頁都包含相關程式碼和詳細説明。
Model
from sentence_transformers import SentenceTransformer, SentenceTransformerModelCardData
model = SentenceTransformer(
"google/embeddinggemma-300m",
model_card_data=SentenceTransformerModelCardData(
language="en",
license="apache-2.0",
model_name="EmbeddingGemma-300m trained on the Medical Instruction and RetrIeval Dataset (MIRIAD)",
),
)
# SentenceTransformer(
# (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
# (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
# (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
# (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
# (4): Normalize()
# )
此程式碼會從 Hugging Face 載入 EmbeddingGemma 模型,並可選擇加入模型卡中繼資料,方便記錄和分享。SentenceTransformer 類別會載入模型權重和設定;model_card_data 參數則會附加中繼資料,以便納入自動生成的模型卡。
資料集
from datasets import load_dataset
train_dataset = load_dataset("tomaarsen/miriad-4.4M-split", split="train").select(range(100_000))
eval_dataset = load_dataset("tomaarsen/miriad-4.4M-split", split="eval").select(range(1_000))
test_dataset = load_dataset("tomaarsen/miriad-4.4M-split", split="test").select(range(1_000))
# Dataset({
# features: ['question', 'passage_text'],
# num_rows: 100000
# })
# Dataset({
# features: ['question', 'passage_text'],
# num_rows: 1000
# })
# Dataset({
# features: ['question', 'passage_text'],
# num_rows: 1000
# })
此程式碼會載入 MIRIAD 資料集,更準確來説,是一份已分成訓練、評估和測試分割的副本。使用大型高品質資料集能確保模型學到有意義的表徵;選取部分樣本則可加快實驗。load_dataset 函式會從 Hugging Face Datasets 取得資料集,.select() 方法則限制各分割使用的樣本數。
損失函式
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=8)
此程式碼使用 Cached Multiple Negatives Ranking Loss(CMNRL)定義訓練損失函式。CMNRL 適合檢索任務,因為它使用批次內負例,高效率訓練模型區分正確和錯誤配對。此損失函式接收問答配對,並將批次中的其他答案視為負例,擴大嵌入空間中不相關配對的距離。mini_batch_size 參數會影響記憶體用量,但不影響訓練動態。
建議在 per_device_train_batch_size 參數採用較大值(該參數屬於 SentenceTransformerTrainingArguments),並在 mini_batch_size 採用較小值(屬於 CachedMultipleNegativesRankingLoss),以低記憶體用量取得強訓練訊號。此外,建議使用 NO_DUPLICATES 批次取樣器,避免意外產生假負例。
訓練參數
from sentence_transformers.training_args import BatchSamplers
from sentence_transformers import SentenceTransformerTrainingArguments
run_name = "embeddinggemma-300m-medical-100k"
args = SentenceTransformerTrainingArguments(
output_dir=f"models/{run_name}",
num_train_epochs=1,
per_device_train_batch_size=128,
per_device_eval_batch_size=128,
learning_rate=2e-5,
warmup_ratio=0.1,
fp16=True, # Set to False if your GPU can't run FP16
bf16=False, # Set to True if your GPU supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES,
prompts={
"question": model.prompts["query"],
"passage_text": model.prompts["document"],
},
eval_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100,
save_total_limit=2,
logging_steps=20,
run_name=run_name,
)
此程式碼會設定訓練、評估和記錄所需的超參數及設定。適當的訓練參數對高效率、穩定且可重現的訓練至關重要。這些參數控制批次大小、學習率、混合精度、評估和儲存頻率等。值得注意的是,prompts 字典會將資料集欄位映射至模型用來區分查詢和文件的提示。
評估器
from sentence_transformers.evaluation import InformationRetrievalEvaluator
queries = dict(enumerate(eval_dataset["question"]))
corpus = dict(enumerate(eval_dataset["passage_text"] + train_dataset["passage_text"][:30_000]))
relevant_docs = {idx: [idx] for idx in queries}
dev_evaluator = InformationRetrievalEvaluator(
queries=queries,
corpus=corpus,
relevant_docs=relevant_docs,
name="miriad-eval-1kq-31kd",
show_progress_bar=True,
)
dev_evaluator(model)
此程式碼會設定資訊檢索評估器,並使用查詢和語料庫測量模型表現。訓練期間進行評估,有助監察進度並避免過度擬合。評估器會檢查模型是否為每個查詢檢索到正確段落,並計算檢索指標(NDCG、MRR、Recall、Precision、MAP 等)。可在訓練前、期間和之後執行,結果會記錄並納入自動生成的模型卡。
請注意,此程式碼片段特別使用全部 1k 個評估問題,對照全部 1k 個評估段落和 30k 個訓練段落,共計 31k 份文件。若只對照評估段落,對模型而言太簡單。
訓練器
from sentence_transformers import SentenceTransformerTrainer
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
evaluator=dev_evaluator,
)
trainer.train()
此程式碼會初始化並執行訓練循環,協調所有組件。
完整微調腳本
以下是結合上述所有組件的完整腳本:
import logging
import traceback
from datasets import load_dataset
from sentence_transformers import (
SentenceTransformer,
SentenceTransformerModelCardData,
SentenceTransformerTrainer,
SentenceTransformerTrainingArguments,
)
from sentence_transformers.evaluation import InformationRetrievalEvaluator
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
from sentence_transformers.training_args import BatchSamplers
# Set the log level to INFO to get more information
logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
# 1. Load a model to finetune with 2. (Optional) model card data
model = SentenceTransformer(
"google/embeddinggemma-300m",
model_card_data=SentenceTransformerModelCardData(
language="en",
license="apache-2.0",
model_name="EmbeddingGemma-300m trained on the Medical Instruction and RetrIeval Dataset (MIRIAD)",
),
)
# 3. Load a dataset to finetune on
train_dataset = load_dataset("tomaarsen/miriad-4.4M-split", split="train").select(range(100_000))
eval_dataset = load_dataset("tomaarsen/miriad-4.4M-split", split="eval").select(range(1_000))
test_dataset = load_dataset("tomaarsen/miriad-4.4M-split", split="test").select(range(1_000))
# 4. Define a loss function. CachedMultipleNegativesRankingLoss (CMNRL) is a special variant of MNRL (a.k.a. InfoNCE),
# which take question-answer pairs (or triplets, etc.) as input. It will take answers from other questions in the batch
# as wrong answers, reducing the distance between the question and the true answer while increasing the distance to the
# wrong answers, in the embedding space.
# The (C)MNRL losses benefit from larger `per_device_train_batch_size` in the Training Arguments, as they can leverage
# more in-batch negative samples. At the same time, the `mini_batch_size` does not affect training performance, but it
# does limit the memory usage. A good trick is setting a high `per_device_train_batch_size` while keeping
# `mini_batch_size` small.
loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=8)
# 5. (Optional) Specify training arguments
run_name = "embeddinggemma-300m-medical-100k"
args = SentenceTransformerTrainingArguments(
# Required parameter:
output_dir=f"models/{run_name}",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=128,
per_device_eval_batch_size=128,
learning_rate=2e-5,
warmup_ratio=0.1,
fp16=True, # Set to False if you get an error that your GPU can't run on FP16
bf16=False, # Set to True if you have a GPU that supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES, # (Cached)MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
prompts={ # Map training column names to model prompts
"question": model.prompts["query"],
"passage_text": model.prompts["document"],
},
# Optional tracking/debugging parameters:
eval_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100,
save_total_limit=2,
logging_steps=20,
run_name=run_name, # Will be used in W&B if `wandb` is installed
)
# 6. (Optional) Create an evaluator using the evaluation queries and 31k answers & evaluate the base model
queries = dict(enumerate(eval_dataset["question"]))
corpus = dict(enumerate(eval_dataset["passage_text"] + train_dataset["passage_text"][:30_000]))
relevant_docs = {idx: [idx] for idx in queries}
dev_evaluator = InformationRetrievalEvaluator(
queries=queries,
corpus=corpus,
relevant_docs=relevant_docs,
name="miriad-eval-1kq-31kd", # 1k questions, 31k passages
show_progress_bar=True,
)
dev_evaluator(model)
# 7. Create a trainer & train
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
evaluator=dev_evaluator,
)
trainer.train()
# (Optional) Evaluate the trained model on the evaluation set once more, this will also log the results
# and include them in the model card
dev_evaluator(model)
queries = dict(enumerate(test_dataset["question"]))
corpus = dict(enumerate(test_dataset["passage_text"] + train_dataset["passage_text"][:30_000]))
relevant_docs = {idx: [idx] for idx in queries}
test_evaluator = InformationRetrievalEvaluator(
queries=queries,
corpus=corpus,
relevant_docs=relevant_docs,
name="miriad-test-1kq-31kd", # 1k questions, 31k passages
show_progress_bar=True,
)
test_evaluator(model)
# 8. Save the trained model
final_output_dir = f"models/{run_name}/final"
model.save_pretrained(final_output_dir)
# 9. (Optional) Push it to the Hugging Face Hub
# It is recommended to run `huggingface-cli login` to log into your Hugging Face account first
try:
model.push_to_hub(run_name)
except Exception:
logging.error(
f"Error uploading model to the Hugging Face Hub:\n{traceback.format_exc()}To upload it manually, you can run "
f"`huggingface-cli login`, followed by loading the model using `model = SentenceTransformer({final_output_dir!r})` "
f"and saving it using `model.push_to_hub('{run_name}')`."
)
訓練
我們在配備 24GB VRAM 的 RTX 3090 上執行完整訓練腳本,訓練和評估共花了 5.5 小時。如要進一步降低記憶體佔用,可減少 mini_batch_size(屬於 CachedMultipleNegativesRankingLoss),以及 batch_size(屬於 InformationRetrievalEvaluator 實例)。以下是我們訓練時的日誌:
| Epoch | 步數 | 訓練損失 | 驗證損失 | miriad-eval-1kq-31kd_cosine_ndcg@10 | miriad-test-1kq-31kd_cosine_ndcg@10 |
|---|---|---|---|---|---|
| -1 | -1 | - | - | 0.8474 | 0.8340 |
| 0.0256 | 20 | 0.1019 | - | - | - |
| 0.0512 | 40 | 0.0444 | - | - | - |
| 0.0767 | 60 | 0.0408 | - | - | - |
| 0.1023 | 80 | 0.0462 | - | - | - |
| 0.1279 | 100 | 0.0542 | 0.0525 | 0.8616 | - |
| 0.1535 | 120 | 0.0454 | - | - | - |
| 0.1790 | 140 | 0.0403 | - | - | - |
| 0.2046 | 160 | 0.0463 | - | - | - |
| 0.2302 | 180 | 0.0508 | - | - | - |
| 0.2558 | 200 | 0.0497 | 0.0449 | 0.8643 | - |
| 0.2813 | 220 | 0.0451 | - | - | - |
| 0.3069 | 240 | 0.0445 | - | - | - |
| 0.3325 | 260 | 0.0489 | - | - | - |
| 0.3581 | 280 | 0.0452 | - | - | - |
| 0.3836 | 300 | 0.0461 | 0.0406 | 0.8832 | - |
| 0.4092 | 320 | 0.0415 | - | - | - |
| 0.4348 | 340 | 0.04 | - | - | - |
| 0.4604 | 360 | 0.0399 | - | - | - |
| 0.4859 | 380 | 0.0423 | - | - | - |
| 0.5115 | 400 | 0.0352 | 0.0316 | 0.8823 | - |
| 0.5371 | 420 | 0.0408 | - | - | - |
| 0.5627 | 440 | 0.0356 | - | - | - |
| 0.5882 | 460 | 0.0371 | - | - | - |
| 0.6138 | 480 | 0.0276 | - | - | - |
| 0.6394 | 500 | 0.028 | 0.0280 | 0.8807 | - |
| 0.6650 | 520 | 0.0302 | - | - | - |
| 0.6905 | 540 | 0.0345 | - | - | - |
| 0.7161 | 560 | 0.0325 | - | - | - |
| 0.7417 | 580 | 0.033 | - | - | - |
| 0.7673 | 600 | 0.0314 | 0.0264 | 0.8910 | - |
| 0.7928 | 620 | 0.033 | - | - | - |
| 0.8184 | 640 | 0.029 | - | - | - |
| 0.8440 | 660 | 0.0396 | - | - | - |
| 0.8696 | 680 | 0.0266 | - | - | - |
| 0.8951 | 700 | 0.0262 | 0.0240 | 0.8968 | - |
| 0.9207 | 720 | 0.0262 | - | - | - |
| 0.9463 | 740 | 0.0327 | - | - | - |
| 0.9719 | 760 | 0.0293 | - | - | - |
| 0.9974 | 780 | 0.0304 | - | - | - |
| -1 | -1 | - | - | 0.9026 | 0.8862 |
微調後評估
基礎模型的表現已經很出色,在我們的 MIRIAD 測試集上取得 0.8340 NDCG@10。即使如此,我們仍能在這個特定領域資料集上大幅提升表現。
| 模型 | 參數數量 | miriad-test-1kq-31kd 上的 NDCG@10 |
|---|---|---|
BAAI/bge-base-en-v1.5 |
109M | 0.7541 |
intfloat/multilingual-e5-small |
118M | 0.6852 |
ibm-granite/granite-embedding-125m-english |
125M | 0.7745 |
Snowflake/snowflake-arctic-embed-m-long |
137M | 0.7514 |
intfloat/multilingual-e5-base |
278M | 0.7052 |
Snowflake/snowflake-arctic-embed-m-v2.0 |
305M | 0.8467 |
BAAI/bge-large-en-v1.5 |
335M | 0.7727 |
mixedbread-ai/mxbai-embed-large-v1 |
335M | 0.7851 |
intfloat/multilingual-e5-large |
560M | 0.7318 |
Snowflake/snowflake-arctic-embed-l-v2.0 |
568M | 0.8433 |
Qwen/Qwen3-Embedding-0.6B |
596M | 0.8493 |
google/embeddinggemma-300m(基礎模型) |
268M | 0.8340 |
sentence-transformers/embeddinggemma-300m-medical(微調後) |
268M | 0.8862 |
我們的微調流程令測試集 NDCG@10 顯著提升 +0.0522,所得模型在目前的模型規模下,於此特定任務的表現輕鬆勝過任何現有通用嵌入模型。若投入更多時間和運算資源,還可取得更佳結果,例如挖掘困難負例,或使用超過 100k 個資料配對訓練。
延伸閲讀
- google/embeddinggemma-300m
- Google EmbeddingGemma 網誌文章
- Google EmbeddingGemma 技術報告
- Sentence Transformers 文件
- Sentence Transformers > 訓練概覽文件
- Transformers.js 文件
- Text Embeddings Inference(TEI)文件
Sentence Transformers 相關網誌文章
如要延伸本文的微調方法,或將相關技術套用至 EmbeddingGemma 和其他嵌入模型:
- 使用 Sentence Transformers 訓練和微調嵌入模型:本文微調章節所依據的端到端訓練指南。
- 使用 Sentence Transformers 訓練和微調重排序模型:Cross Encoder 訓練,常作為 EmbeddingGemma 檢索器之後的第二階段。
- 使用 Sentence Transformers 訓練和微調稀疏嵌入模型:SPLADE 訓練,可與 EmbeddingGemma 配合進行混合檢索。
- 🪆 Matryoshka 嵌入模型簡介:瞭解 EmbeddingGemma 可截短嵌入的背景。
- 二元和純量嵌入量化,大幅加快並降低檢索成本:在檢索期間進一步縮減 EmbeddingGemma 嵌入。
來源:Hugging Face Blog · huggingface.co