Transformer架构深度解析：现代AI的基石

✨“俺はモンキー・D・ルフィ。海贼王になる男だ！”

• • 引言
  • 为什么需要Transformer？
  • 整体架构
  • 核心组件详解
  • • 1. 自注意力机制（Self-Attention）
    • • 计算公式
      • 代码实现
    • 2. 多头注意力（Multi-Head Attention）
    • 3. 位置编码（Positional Encoding）
  • 完整Transformer实现
  • Transformer处理流程
  • 实际应用示例：机器翻译
  • Transformer家族
  • 总结
  • 进一步学习

引言

2017年，Google团队发表了划时代论文《Attention Is All You Need》，提出了Transformer架构。这个架构彻底改变了自然语言处理领域，成为了ChatGPT、BERT等大模型的基础。

为什么需要Transformer？

在Transformer之前，RNN和LSTM是处理序列数据的主流模型，但它们存在两个主要问题：

Transformer通过自注意力机制解决了这些问题。

整体架构

#mermaid-svg-KB2s50txRGfWh2Om{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-KB2s50txRGfWh2Om .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-KB2s50txRGfWh2Om .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-KB2s50txRGfWh2Om .error-icon{fill:#552222;}#mermaid-svg-KB2s50txRGfWh2Om .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-KB2s50txRGfWh2Om .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-KB2s50txRGfWh2Om .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-KB2s50txRGfWh2Om .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-KB2s50txRGfWh2Om .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-KB2s50txRGfWh2Om .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-KB2s50txRGfWh2Om .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-KB2s50txRGfWh2Om .marker{fill:#333333;stroke:#333333;}#mermaid-svg-KB2s50txRGfWh2Om .marker.cross{stroke:#333333;}#mermaid-svg-KB2s50txRGfWh2Om svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-KB2s50txRGfWh2Om p{margin:0;}#mermaid-svg-KB2s50txRGfWh2Om .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-KB2s50txRGfWh2Om .cluster-label text{fill:#333;}#mermaid-svg-KB2s50txRGfWh2Om .cluster-label span{color:#333;}#mermaid-svg-KB2s50txRGfWh2Om .cluster-label span p{background-color:transparent;}#mermaid-svg-KB2s50txRGfWh2Om .label text,#mermaid-svg-KB2s50txRGfWh2Om span{fill:#333;color:#333;}#mermaid-svg-KB2s50txRGfWh2Om .node rect,#mermaid-svg-KB2s50txRGfWh2Om .node circle,#mermaid-svg-KB2s50txRGfWh2Om .node ellipse,#mermaid-svg-KB2s50txRGfWh2Om .node polygon,#mermaid-svg-KB2s50txRGfWh2Om .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-KB2s50txRGfWh2Om .rough-node .label text,#mermaid-svg-KB2s50txRGfWh2Om .node .label text,#mermaid-svg-KB2s50txRGfWh2Om .image-shape .label,#mermaid-svg-KB2s50txRGfWh2Om .icon-shape .label{text-anchor:middle;}#mermaid-svg-KB2s50txRGfWh2Om .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-KB2s50txRGfWh2Om .rough-node .label,#mermaid-svg-KB2s50txRGfWh2Om .node .label,#mermaid-svg-KB2s50txRGfWh2Om .image-shape .label,#mermaid-svg-KB2s50txRGfWh2Om .icon-shape .label{text-align:center;}#mermaid-svg-KB2s50txRGfWh2Om .node.clickable{cursor:pointer;}#mermaid-svg-KB2s50txRGfWh2Om .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-KB2s50txRGfWh2Om .arrowheadPath{fill:#333333;}#mermaid-svg-KB2s50txRGfWh2Om .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-KB2s50txRGfWh2Om .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-KB2s50txRGfWh2Om .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KB2s50txRGfWh2Om .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-KB2s50txRGfWh2Om .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KB2s50txRGfWh2Om .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-KB2s50txRGfWh2Om .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-KB2s50txRGfWh2Om .cluster text{fill:#333;}#mermaid-svg-KB2s50txRGfWh2Om .cluster span{color:#333;}#mermaid-svg-KB2s50txRGfWh2Om div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-KB2s50txRGfWh2Om .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-KB2s50txRGfWh2Om rect.text{fill:none;stroke-width:0;}#mermaid-svg-KB2s50txRGfWh2Om .icon-shape,#mermaid-svg-KB2s50txRGfWh2Om .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KB2s50txRGfWh2Om .icon-shape p,#mermaid-svg-KB2s50txRGfWh2Om .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-KB2s50txRGfWh2Om .icon-shape rect,#mermaid-svg-KB2s50txRGfWh2Om .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KB2s50txRGfWh2Om .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-KB2s50txRGfWh2Om .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-KB2s50txRGfWh2Om :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

解码器

编码器

Self-Attention层

Feed Forward层

Masked Self-Attention

Cross-Attention

Feed Forward层

输入嵌入

输出嵌入

线性+Softmax

核心组件详解

1. 自注意力机制（Self-Attention）

自注意力是Transformer的核心，让模型能够关注序列中不同位置的关系。

#mermaid-svg-YqEismdjmRFfkFu7{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-YqEismdjmRFfkFu7 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-YqEismdjmRFfkFu7 .error-icon{fill:#552222;}#mermaid-svg-YqEismdjmRFfkFu7 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-YqEismdjmRFfkFu7 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-YqEismdjmRFfkFu7 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-YqEismdjmRFfkFu7 .marker.cross{stroke:#333333;}#mermaid-svg-YqEismdjmRFfkFu7 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-YqEismdjmRFfkFu7 p{margin:0;}#mermaid-svg-YqEismdjmRFfkFu7 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-YqEismdjmRFfkFu7 .cluster-label text{fill:#333;}#mermaid-svg-YqEismdjmRFfkFu7 .cluster-label span{color:#333;}#mermaid-svg-YqEismdjmRFfkFu7 .cluster-label span p{background-color:transparent;}#mermaid-svg-YqEismdjmRFfkFu7 .label text,#mermaid-svg-YqEismdjmRFfkFu7 span{fill:#333;color:#333;}#mermaid-svg-YqEismdjmRFfkFu7 .node rect,#mermaid-svg-YqEismdjmRFfkFu7 .node circle,#mermaid-svg-YqEismdjmRFfkFu7 .node ellipse,#mermaid-svg-YqEismdjmRFfkFu7 .node polygon,#mermaid-svg-YqEismdjmRFfkFu7 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-YqEismdjmRFfkFu7 .rough-node .label text,#mermaid-svg-YqEismdjmRFfkFu7 .node .label text,#mermaid-svg-YqEismdjmRFfkFu7 .image-shape .label,#mermaid-svg-YqEismdjmRFfkFu7 .icon-shape .label{text-anchor:middle;}#mermaid-svg-YqEismdjmRFfkFu7 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-YqEismdjmRFfkFu7 .rough-node .label,#mermaid-svg-YqEismdjmRFfkFu7 .node .label,#mermaid-svg-YqEismdjmRFfkFu7 .image-shape .label,#mermaid-svg-YqEismdjmRFfkFu7 .icon-shape .label{text-align:center;}#mermaid-svg-YqEismdjmRFfkFu7 .node.clickable{cursor:pointer;}#mermaid-svg-YqEismdjmRFfkFu7 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-YqEismdjmRFfkFu7 .arrowheadPath{fill:#333333;}#mermaid-svg-YqEismdjmRFfkFu7 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-YqEismdjmRFfkFu7 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-YqEismdjmRFfkFu7 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-YqEismdjmRFfkFu7 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-YqEismdjmRFfkFu7 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-YqEismdjmRFfkFu7 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-YqEismdjmRFfkFu7 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-YqEismdjmRFfkFu7 .cluster text{fill:#333;}#mermaid-svg-YqEismdjmRFfkFu7 .cluster span{color:#333;}#mermaid-svg-YqEismdjmRFfkFu7 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-YqEismdjmRFfkFu7 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-YqEismdjmRFfkFu7 rect.text{fill:none;stroke-width:0;}#mermaid-svg-YqEismdjmRFfkFu7 .icon-shape,#mermaid-svg-YqEismdjmRFfkFu7 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-YqEismdjmRFfkFu7 .icon-shape p,#mermaid-svg-YqEismdjmRFfkFu7 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-YqEismdjmRFfkFu7 .icon-shape rect,#mermaid-svg-YqEismdjmRFfkFu7 .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-YqEismdjmRFfkFu7 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-YqEismdjmRFfkFu7 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-YqEismdjmRFfkFu7 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

输入向量 X

生成 Q, K, V

计算 Attention Score

缩放点积

Softmax归一化

加权求和

计算公式

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

• Q（Query）：查询向量，表示要找什么
• K（Key）：键向量，表示被查询的内容
• V（Value）：值向量，表示实际内容

代码实现

import torch
import torch.nn as nn
import torch.nn.functional as F

class SelfAttention(nn.Module):
def __init__(self, embed_size, heads):
super(SelfAttention, self).__init__()
self.embed_size = embed_size
self.heads = heads
self.head_dim = embed_size // heads

# 线性变换生成Q, K, V
self.values = nn.Linear(self.head_dim, self.head_dim, bias=False)
self.keys = nn.Linear(self.head_dim, self.head_dim, bias=False)
self.queries = nn.Linear(self.head_dim, self.head_dim, bias=False)

# 输出投影
self.fc_out = nn.Linear(heads * self.head_dim, embed_size)

def forward(self, values, keys, query, mask):
N = query.shape[0] # batch size
value_len, key_len, query_len = values.shape[1], keys.shape[1], query.shape[1]

# 分割成多个头
values = values.reshape(N, value_len, self.heads, self.head_dim)
keys = keys.reshape(N, key_len, self.heads, self.head_dim)
queries = query.reshape(N, query_len, self.heads, self.head_dim)

# 计算注意力分数
energy = torch.einsum("nqhd,nkhd->nhqk", [queries, keys])

# 缩放
energy = energy / (self.embed_size ** (1/2))

# Mask（可选，用于解码器）
if mask is not None:
energy = energy.masked_fill(mask == 0, float("-1e20"))

# Softmax
attention = torch.softmax(energy, dim=3)

# 加权求和
out = torch.einsum("nhql,nlhd->nqhd", [attention, values])

# 拼接所有头
out = out.reshape(N, query_len, self.heads * self.head_dim)

return self.fc_out(out)

2. 多头注意力（Multi-Head Attention）

多头注意力让模型从不同角度理解输入：

#mermaid-svg-FZLYigtFdQyQsZDY{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-FZLYigtFdQyQsZDY .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-FZLYigtFdQyQsZDY .error-icon{fill:#552222;}#mermaid-svg-FZLYigtFdQyQsZDY .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-FZLYigtFdQyQsZDY .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-FZLYigtFdQyQsZDY .marker{fill:#333333;stroke:#333333;}#mermaid-svg-FZLYigtFdQyQsZDY .marker.cross{stroke:#333333;}#mermaid-svg-FZLYigtFdQyQsZDY svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-FZLYigtFdQyQsZDY p{margin:0;}#mermaid-svg-FZLYigtFdQyQsZDY .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-FZLYigtFdQyQsZDY .cluster-label text{fill:#333;}#mermaid-svg-FZLYigtFdQyQsZDY .cluster-label span{color:#333;}#mermaid-svg-FZLYigtFdQyQsZDY .cluster-label span p{background-color:transparent;}#mermaid-svg-FZLYigtFdQyQsZDY .label text,#mermaid-svg-FZLYigtFdQyQsZDY span{fill:#333;color:#333;}#mermaid-svg-FZLYigtFdQyQsZDY .node rect,#mermaid-svg-FZLYigtFdQyQsZDY .node circle,#mermaid-svg-FZLYigtFdQyQsZDY .node ellipse,#mermaid-svg-FZLYigtFdQyQsZDY .node polygon,#mermaid-svg-FZLYigtFdQyQsZDY .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-FZLYigtFdQyQsZDY .rough-node .label text,#mermaid-svg-FZLYigtFdQyQsZDY .node .label text,#mermaid-svg-FZLYigtFdQyQsZDY .image-shape .label,#mermaid-svg-FZLYigtFdQyQsZDY .icon-shape .label{text-anchor:middle;}#mermaid-svg-FZLYigtFdQyQsZDY .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-FZLYigtFdQyQsZDY .rough-node .label,#mermaid-svg-FZLYigtFdQyQsZDY .node .label,#mermaid-svg-FZLYigtFdQyQsZDY .image-shape .label,#mermaid-svg-FZLYigtFdQyQsZDY .icon-shape .label{text-align:center;}#mermaid-svg-FZLYigtFdQyQsZDY .node.clickable{cursor:pointer;}#mermaid-svg-FZLYigtFdQyQsZDY .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-FZLYigtFdQyQsZDY .arrowheadPath{fill:#333333;}#mermaid-svg-FZLYigtFdQyQsZDY .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-FZLYigtFdQyQsZDY .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-FZLYigtFdQyQsZDY .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FZLYigtFdQyQsZDY .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-FZLYigtFdQyQsZDY .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FZLYigtFdQyQsZDY .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-FZLYigtFdQyQsZDY .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-FZLYigtFdQyQsZDY .cluster text{fill:#333;}#mermaid-svg-FZLYigtFdQyQsZDY .cluster span{color:#333;}#mermaid-svg-FZLYigtFdQyQsZDY div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-FZLYigtFdQyQsZDY .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-FZLYigtFdQyQsZDY rect.text{fill:none;stroke-width:0;}#mermaid-svg-FZLYigtFdQyQsZDY .icon-shape,#mermaid-svg-FZLYigtFdQyQsZDY .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FZLYigtFdQyQsZDY .icon-shape p,#mermaid-svg-FZLYigtFdQyQsZDY .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-FZLYigtFdQyQsZDY .icon-shape rect,#mermaid-svg-FZLYigtFdQyQsZDY .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FZLYigtFdQyQsZDY .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-FZLYigtFdQyQsZDY .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-FZLYigtFdQyQsZDY :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

输入

头1: 关注语法

头2: 关注语义

头3: 关注上下文

头4: 关注情感

拼接

线性变换

class TransformerBlock(nn.Module):
def __init__(self, embed_size, heads, dropout, forward_expansion):
super(TransformerBlock, self).__init__()
self.attention = SelfAttention(embed_size, heads)
self.norm1 = nn.LayerNorm(embed_size)
self.norm2 = nn.LayerNorm(embed_size)

self.feed_forward = nn.Sequential(
nn.Linear(embed_size, forward_expansion * embed_size),
nn.ReLU(),
nn.Linear(forward_expansion * embed_size, embed_size)
)

self.dropout = nn.Dropout(dropout)

def forward(self, value, key, query, mask):
# 多头注意力 + 残差连接
attention = self.attention(value, key, query, mask)
x = self.dropout(self.norm1(attention + query))

# 前馈网络 + 残差连接
forward = self.feed_forward(x)
out = self.dropout(self.norm2(forward + x))
return out

3. 位置编码（Positional Encoding）

由于注意力机制没有顺序概念，需要位置编码注入位置信息：

class PositionalEncoding(nn.Module):
def __init__(self, embed_size, max_len=5000):
super(PositionalEncoding, self).__init__()

# 创建位置编码矩阵
pe = torch.zeros(max_len, embed_size)
position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
div_term = torch.exp(torch.arange(0, embed_size, 2).float() *
(–math.log(10000.0) / embed_size))

pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
pe = pe.unsqueeze(0)

# 注册为buffer（不是参数，但会保存到模型）
self.register_buffer('pe', pe)

def forward(self, x):
# x shape: [batch_size, seq_len, embed_size]
return x + self.pe[:, :x.size(1)]

完整Transformer实现

import math
import torch
import torch.nn as nn

class Transformer(nn.Module):
def __init__(self, src_vocab_size, trg_vocab_size, embed_size=512,
num_layers=6, heads=8, forward_expansion=4, dropout=0.1):
super(Transformer, self).__init__()

# 词嵌入
self.src_word_embedding = nn.Embedding(src_vocab_size, embed_size)
self.trg_word_embedding = nn.Embedding(trg_vocab_size, embed_size)

# 位置编码
self.position_encoding = PositionalEncoding(embed_size)

# 编码器层
self.encoder = nn.ModuleList([
TransformerBlock(embed_size, heads, dropout, forward_expansion)
for _ in range(num_layers)
])

# 解码器层
self.decoder = nn.ModuleList([
TransformerBlock(embed_size, heads, dropout, forward_expansion)
for _ in range(num_layers)
])

# 输出层
self.fc_out = nn.Linear(embed_size, trg_vocab_size)
self.dropout = nn.Dropout(dropout)

def forward(self, src, trg, src_mask, trg_mask):
# 嵌入 + 位置编码
src_embed = self.dropout(self.position_encoding(self.src_word_embedding(src)))
trg_embed = self.dropout(self.position_encoding(self.trg_word_embedding(trg)))

# 编码器
enc_output = src_embed
for layer in self.encoder:
enc_output = layer(enc_output, enc_output, enc_output, src_mask)

# 解码器
dec_output = trg_embed
for layer in self.decoder:
dec_output = layer(dec_output, enc_output, dec_output, trg_mask)

# 输出预测
out = self.fc_out(dec_output)
return out

Transformer处理流程

输出概率

解码器×N层

编码器×N层

词嵌入+位置编码

输入序列

输出概率

解码器×N层

编码器×N层

词嵌入+位置编码

输入序列

#mermaid-svg-oVRcGoryGqrPlZfv{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-oVRcGoryGqrPlZfv .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-oVRcGoryGqrPlZfv .error-icon{fill:#552222;}#mermaid-svg-oVRcGoryGqrPlZfv .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-oVRcGoryGqrPlZfv .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-oVRcGoryGqrPlZfv .marker{fill:#333333;stroke:#333333;}#mermaid-svg-oVRcGoryGqrPlZfv .marker.cross{stroke:#333333;}#mermaid-svg-oVRcGoryGqrPlZfv svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-oVRcGoryGqrPlZfv p{margin:0;}#mermaid-svg-oVRcGoryGqrPlZfv .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-oVRcGoryGqrPlZfv text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-oVRcGoryGqrPlZfv .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-oVRcGoryGqrPlZfv .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-oVRcGoryGqrPlZfv .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-oVRcGoryGqrPlZfv .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-oVRcGoryGqrPlZfv #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-oVRcGoryGqrPlZfv .sequenceNumber{fill:white;}#mermaid-svg-oVRcGoryGqrPlZfv #sequencenumber{fill:#333;}#mermaid-svg-oVRcGoryGqrPlZfv #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-oVRcGoryGqrPlZfv .messageText{fill:#333;stroke:none;}#mermaid-svg-oVRcGoryGqrPlZfv .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-oVRcGoryGqrPlZfv .labelText,#mermaid-svg-oVRcGoryGqrPlZfv .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-oVRcGoryGqrPlZfv .loopText,#mermaid-svg-oVRcGoryGqrPlZfv .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-oVRcGoryGqrPlZfv .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-oVRcGoryGqrPlZfv .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-oVRcGoryGqrPlZfv .noteText,#mermaid-svg-oVRcGoryGqrPlZfv .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-oVRcGoryGqrPlZfv .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-oVRcGoryGqrPlZfv .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-oVRcGoryGqrPlZfv .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-oVRcGoryGqrPlZfv .actorPopupMenu{position:absolute;}#mermaid-svg-oVRcGoryGqrPlZfv .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-oVRcGoryGqrPlZfv .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-oVRcGoryGqrPlZfv .actor-man circle,#mermaid-svg-oVRcGoryGqrPlZfv line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-oVRcGoryGqrPlZfv :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Token IDs

添加位置信息

编码后的向量

多层注意力处理

编码器输出(作为K,V)

自注意力+交叉注意力

预测下一个词

实际应用示例：机器翻译

# 简化的翻译模型使用示例
def translate_sentence(model, sentence, src_vocab, trg_vocab, device, max_length=50):
model.eval()

# 分词并转换为索引
tokens = [src_vocab.get(token, src_vocab['<unk>']) for token in sentence.split()]
tokens = torch.LongTensor(tokens).unsqueeze(0).to(device)

# 创建目标序列（从<sos>开始）
trg_tensor = torch.LongTensor([[trg_vocab['<sos>']]]).to(device)

with torch.no_grad():
for _ in range(max_length):
# 生成mask
trg_mask = (trg_tensor != trg_vocab['<pad>']).unsqueeze(1)

# 前向传播
output = model(tokens, trg_tensor, src_mask=None, trg_mask=trg_mask)

# 获取预测的下一个词
pred_token = output.argmax(2)[:, –1].item()

# 如果是结束符，停止
if pred_token == trg_vocab['<eos>']:
break

# 添加到输出序列
trg_tensor = torch.cat([trg_tensor,
torch.LongTensor([[pred_token]]).to(device)], dim=1)

return trg_tensor

Transformer家族

#mermaid-svg-JUe6V4ZPlWWjAxRn{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-JUe6V4ZPlWWjAxRn .error-icon{fill:#552222;}#mermaid-svg-JUe6V4ZPlWWjAxRn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-JUe6V4ZPlWWjAxRn .marker{fill:#333333;stroke:#333333;}#mermaid-svg-JUe6V4ZPlWWjAxRn .marker.cross{stroke:#333333;}#mermaid-svg-JUe6V4ZPlWWjAxRn svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-JUe6V4ZPlWWjAxRn p{margin:0;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge{stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 text{fill:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth–1{stroke-width:17;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-0{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-0{stroke-width:14;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-1{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-1{stroke-width:11;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 text{fill:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-2{stroke-width:8;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-3{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-3{stroke-width:5;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-4{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-4{stroke-width:2;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-5{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-5{stroke-width:-1;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-6{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-6{stroke-width:-4;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-7{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-7{stroke-width:-7;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-8{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-8{stroke-width:-10;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-9{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-9{stroke-width:-13;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 polygon,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 text{fill:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .node-icon-10{font-size:40px;color:black;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge-depth-10{stroke-width:-16;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:lightgray;}#mermaid-svg-JUe6V4ZPlWWjAxRn .disabled text{fill:#efefef;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-root rect,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-root path,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-root circle,#mermaid-svg-JUe6V4ZPlWWjAxRn .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-root text{fill:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-root span{color:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .section-2 span{color:#ffffff;}#mermaid-svg-JUe6V4ZPlWWjAxRn .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-JUe6V4ZPlWWjAxRn .edge{fill:none;}#mermaid-svg-JUe6V4ZPlWWjAxRn .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-JUe6V4ZPlWWjAxRn :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Transformer

编码器为主

BERT

RoBERTa

ALBERT

解码器为主

GPT系列

BLOOM

LLaMA

编解码器

原始Transformer

T5

BART

总结

Transformer的核心优势：

进一步学习

• BERT：双向编码器，适合理解任务
• GPT：自回归解码器，适合生成任务
• ViT：将Transformer应用于计算机视觉
• Efficient Transformer：线性注意力等优化方法

参考资料：

• 《Attention Is All You Need》原论文
• The Illustrated Transformer
• Hugging Face Transformers库文档