跳到正文
DEV Community· DRVEXO_LLM·· 2 天前AI 評分36

Transformer 如何判斷你指的是哪個「bank」?

How does a Transformer know which “bank” you mean?

AI 導讀

Transformer 透過自注意力機制,根據周圍 token 的線索判斷「bank」指河岸還是金融機構。每個 token 的 query 與可見 token 的 key 比較,所得權重決定各 value 對表示的貢獻;多個注意力頭可學習不同關係。原始編碼器-解碼器 Transformer 的解碼器會以因果遮罩排除未來 token,並透過交叉注意力參照編碼後的來源。

正文

DRVEXO_LLM

In “I sat by the bank,” the setting suggests a riverbank. In “I withdrew cash at the bank,” cash points to a financial institution. The spelling stays the same; the surrounding evidence changes.

Self-attention gives each token a query, a key, and a value. The query for “bank” is compared with the keys for the tokens it can see. The resulting weights determine how much each value contributes. Multiple heads can learn different relationships.

In the original encoder–decoder Transformer, the encoder reads the source. The decoder uses causal masking to keep future target tokens out of view and cross-attention to consult the encoded source. Each new token becomes context for the next prediction. The numerical examples in the film are illustrative.

Video: a visual explanation of attention, masking, and decoding.

來源:DEV Community · dev.to