解码双生：Decoder-Only 推理的 Prefill 与 Decode 全景透析

“每一轮生成都是一次精心编排的双幕剧——Prefill 是群舞，Decode 是独步。”

📜 目录

🔬 1. 自回归的本质

Decoder-only 语言模型的核心是自回归生成：每个 token 的生成依赖于之前所有 token。

#mermaid-svg-jS9hxdaQnutCGFSb{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-jS9hxdaQnutCGFSb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-jS9hxdaQnutCGFSb .error-icon{fill:#552222;}#mermaid-svg-jS9hxdaQnutCGFSb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-jS9hxdaQnutCGFSb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-jS9hxdaQnutCGFSb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-jS9hxdaQnutCGFSb .marker.cross{stroke:#333333;}#mermaid-svg-jS9hxdaQnutCGFSb svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-jS9hxdaQnutCGFSb p{margin:0;}#mermaid-svg-jS9hxdaQnutCGFSb .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-jS9hxdaQnutCGFSb .cluster-label text{fill:#333;}#mermaid-svg-jS9hxdaQnutCGFSb .cluster-label span{color:#333;}#mermaid-svg-jS9hxdaQnutCGFSb .cluster-label span p{background-color:transparent;}#mermaid-svg-jS9hxdaQnutCGFSb .label text,#mermaid-svg-jS9hxdaQnutCGFSb span{fill:#333;color:#333;}#mermaid-svg-jS9hxdaQnutCGFSb .node rect,#mermaid-svg-jS9hxdaQnutCGFSb .node circle,#mermaid-svg-jS9hxdaQnutCGFSb .node ellipse,#mermaid-svg-jS9hxdaQnutCGFSb .node polygon,#mermaid-svg-jS9hxdaQnutCGFSb .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-jS9hxdaQnutCGFSb .rough-node .label text,#mermaid-svg-jS9hxdaQnutCGFSb .node .label text,#mermaid-svg-jS9hxdaQnutCGFSb .image-shape .label,#mermaid-svg-jS9hxdaQnutCGFSb .icon-shape .label{text-anchor:middle;}#mermaid-svg-jS9hxdaQnutCGFSb .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-jS9hxdaQnutCGFSb .rough-node .label,#mermaid-svg-jS9hxdaQnutCGFSb .node .label,#mermaid-svg-jS9hxdaQnutCGFSb .image-shape .label,#mermaid-svg-jS9hxdaQnutCGFSb .icon-shape .label{text-align:center;}#mermaid-svg-jS9hxdaQnutCGFSb .node.clickable{cursor:pointer;}#mermaid-svg-jS9hxdaQnutCGFSb .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-jS9hxdaQnutCGFSb .arrowheadPath{fill:#333333;}#mermaid-svg-jS9hxdaQnutCGFSb .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-jS9hxdaQnutCGFSb .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-jS9hxdaQnutCGFSb .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-jS9hxdaQnutCGFSb .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-jS9hxdaQnutCGFSb .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-jS9hxdaQnutCGFSb .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-jS9hxdaQnutCGFSb .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-jS9hxdaQnutCGFSb .cluster text{fill:#333;}#mermaid-svg-jS9hxdaQnutCGFSb .cluster span{color:#333;}#mermaid-svg-jS9hxdaQnutCGFSb div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-jS9hxdaQnutCGFSb .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-jS9hxdaQnutCGFSb rect.text{fill:none;stroke-width:0;}#mermaid-svg-jS9hxdaQnutCGFSb .icon-shape,#mermaid-svg-jS9hxdaQnutCGFSb .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-jS9hxdaQnutCGFSb .icon-shape p,#mermaid-svg-jS9hxdaQnutCGFSb .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-jS9hxdaQnutCGFSb .icon-shape .label rect,#mermaid-svg-jS9hxdaQnutCGFSb .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-jS9hxdaQnutCGFSb .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-jS9hxdaQnutCGFSb .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-jS9hxdaQnutCGFSb :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

P(y₁ | x)

P(y₂ | x, y₁)

P(y₃ | x, y₁, y₂)

…

∏ P(yₜ | x, y₁:t₋₁)

这个链式法则决定了推理本质上存在两个阶段：

⚡ 2. 幕起 · Prefill 阶段深度解构

2.1 定义与边界

Prefill 阶段接收完整的 prompt 序列，利用 Transformer 的并行能力一次性计算所有 token 的 hidden states，同时建立 KV Cache。

#mermaid-svg-TipbdzncasvKqnnp{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-TipbdzncasvKqnnp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-TipbdzncasvKqnnp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-TipbdzncasvKqnnp .error-icon{fill:#552222;}#mermaid-svg-TipbdzncasvKqnnp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-TipbdzncasvKqnnp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-TipbdzncasvKqnnp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-TipbdzncasvKqnnp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-TipbdzncasvKqnnp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-TipbdzncasvKqnnp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-TipbdzncasvKqnnp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-TipbdzncasvKqnnp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-TipbdzncasvKqnnp .marker.cross{stroke:#333333;}#mermaid-svg-TipbdzncasvKqnnp svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-TipbdzncasvKqnnp p{margin:0;}#mermaid-svg-TipbdzncasvKqnnp .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-TipbdzncasvKqnnp .cluster-label text{fill:#333;}#mermaid-svg-TipbdzncasvKqnnp .cluster-label span{color:#333;}#mermaid-svg-TipbdzncasvKqnnp .cluster-label span p{background-color:transparent;}#mermaid-svg-TipbdzncasvKqnnp .label text,#mermaid-svg-TipbdzncasvKqnnp span{fill:#333;color:#333;}#mermaid-svg-TipbdzncasvKqnnp .node rect,#mermaid-svg-TipbdzncasvKqnnp .node circle,#mermaid-svg-TipbdzncasvKqnnp .node ellipse,#mermaid-svg-TipbdzncasvKqnnp .node polygon,#mermaid-svg-TipbdzncasvKqnnp .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-TipbdzncasvKqnnp .rough-node .label text,#mermaid-svg-TipbdzncasvKqnnp .node .label text,#mermaid-svg-TipbdzncasvKqnnp .image-shape .label,#mermaid-svg-TipbdzncasvKqnnp .icon-shape .label{text-anchor:middle;}#mermaid-svg-TipbdzncasvKqnnp .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-TipbdzncasvKqnnp .rough-node .label,#mermaid-svg-TipbdzncasvKqnnp .node .label,#mermaid-svg-TipbdzncasvKqnnp .image-shape .label,#mermaid-svg-TipbdzncasvKqnnp .icon-shape .label{text-align:center;}#mermaid-svg-TipbdzncasvKqnnp .node.clickable{cursor:pointer;}#mermaid-svg-TipbdzncasvKqnnp .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-TipbdzncasvKqnnp .arrowheadPath{fill:#333333;}#mermaid-svg-TipbdzncasvKqnnp .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-TipbdzncasvKqnnp .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-TipbdzncasvKqnnp .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-TipbdzncasvKqnnp .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-TipbdzncasvKqnnp .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-TipbdzncasvKqnnp .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-TipbdzncasvKqnnp .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-TipbdzncasvKqnnp .cluster text{fill:#333;}#mermaid-svg-TipbdzncasvKqnnp .cluster span{color:#333;}#mermaid-svg-TipbdzncasvKqnnp div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-TipbdzncasvKqnnp .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-TipbdzncasvKqnnp rect.text{fill:none;stroke-width:0;}#mermaid-svg-TipbdzncasvKqnnp .icon-shape,#mermaid-svg-TipbdzncasvKqnnp .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-TipbdzncasvKqnnp .icon-shape p,#mermaid-svg-TipbdzncasvKqnnp .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-TipbdzncasvKqnnp .icon-shape .label rect,#mermaid-svg-TipbdzncasvKqnnp .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-TipbdzncasvKqnnp .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-TipbdzncasvKqnnp .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-TipbdzncasvKqnnp :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

逐层并行计算

KV Cache 建立

Layer 1: Kⁿˣᵈ, Vⁿˣᵈ

Layer 2: Kⁿˣᵈ, Vⁿˣᵈ

Layer L: Kⁿˣᵈ, Vⁿˣᵈ

输出 Hidden States

hidden₁

hidden₂

hiddenₙ

输入：用户 Prompt

Token₁

Token₂

…

Tokenₙ

Layer 1 Attention + FFN

Layer 2 Attention + FFN

Layer L Attention + FFN

2.2 数学形式化

设 prompt 长度为 n，embedding 维度为 d，head 数量为 h，每 head 维度为 dₖ = d/h。

对于 Transformer 的第 ℓ 层：

Step 1: 线性投影

Q^ℓ = X^ℓ · W_Q^ℓ # ℝⁿˣᵈ · ℝᵈˣᵈ = ℝⁿˣᵈ
K^ℓ = X^ℓ · W_K^ℓ # ℝⁿˣᵈ · ℝᵈˣᵈ = ℝⁿˣᵈ
V^ℓ = X^ℓ · W_V^ℓ # ℝⁿˣᵈ · ℝᵈˣᵈ = ℝⁿˣᵈ

Step 2: 分头并重塑

Q^ℓ → [Q^ℓ₁, Q^ℓ₂, …, Q^ℓₕ] # 每头 ℝⁿˣᵈₖ
K^ℓ → [K^ℓ₁, K^ℓ₂, …, K^ℓₕ] # 每头 ℝⁿˣᵈₖ
V^ℓ → [V^ℓ₁, V^ℓ₂, …, V^ℓₕ] # 每头 ℝⁿˣᵈₖ

Step 3: Scaled Dot-Product Attention（核心计算）

对于 head i：

S^ℓᵢ = Q^ℓᵢ · (K^ℓᵢ)ᵀ # Score: ℝⁿˣⁿ
A^ℓᵢ = softmax(S^ℓᵢ / √dₖ + M) # 加 causal mask M 后 softmax
O^ℓᵢ = A^ℓᵢ · V^ℓᵢ # 输出: ℝⁿˣᵈₖ

其中 M 是 causal mask：

M_ij = { 0 if i ≥ j (或 j ≤ i，取决于定义)
{ -∞ if i < j

Causal mask 可视化（下三角有效，上三角被掩盖）：

Q₁ Q₂ Q₃ Q₄
K₁ [ 1 -∞ -∞ -∞ ]
K₂ [0.2 1 -∞ -∞ ]
K₃ [0.1 0.3 1 -∞ ]
K₄ [0.05 0.15 0.3 1 ]

每个 token 只能 attend 到自己和之前的 token。

Step 4: 合并多头

O^ℓ = Concat(O^ℓ₁, O^ℓ₂, …, O^ℓₕ) · W_O^ℓ # ℝⁿˣᵈ

Step 5: FFN 与残差连接

X^ℓ₍ₐₜₜ₎ = LayerNorm(X^ℓ + O^ℓ) # Attention 残差
X^{ℓ+1} = LayerNorm(X^ℓ₍ₐₜₜ₎ + FFN(X^ℓ₍ₐₜₜ₎)) # FFN 残差

2.3 计算复杂度分析

Prefill 单层总 FLOPs:
QKV 投影: 3 × (2 × n × d²) = 6nd²
Attention: 2 × (2 × h × n² × dₖ) = 4n²d (QKᵀ + PV)
Attention 后投影: 2 × n × d² = 2nd²
FFN: 2 × (2 × n × d × d_ff) = 4nd·d_ff

总计 ≈ 2n²d + 2n·d_ff·d + O(nd²)

当 n（prompt 长度）很大时，O(n²) 的 Attention 项主导计算量。

2.4 Prefill 的内存行为

#mermaid-svg-FTg6a7lL7dxXt4oa{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-FTg6a7lL7dxXt4oa .error-icon{fill:#552222;}#mermaid-svg-FTg6a7lL7dxXt4oa .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-FTg6a7lL7dxXt4oa .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-FTg6a7lL7dxXt4oa .marker{fill:#333333;stroke:#333333;}#mermaid-svg-FTg6a7lL7dxXt4oa .marker.cross{stroke:#333333;}#mermaid-svg-FTg6a7lL7dxXt4oa svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-FTg6a7lL7dxXt4oa p{margin:0;}#mermaid-svg-FTg6a7lL7dxXt4oa .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa .cluster-label text{fill:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa .cluster-label span{color:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa .cluster-label span p{background-color:transparent;}#mermaid-svg-FTg6a7lL7dxXt4oa .label text,#mermaid-svg-FTg6a7lL7dxXt4oa span{fill:#333;color:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa .node rect,#mermaid-svg-FTg6a7lL7dxXt4oa .node circle,#mermaid-svg-FTg6a7lL7dxXt4oa .node ellipse,#mermaid-svg-FTg6a7lL7dxXt4oa .node polygon,#mermaid-svg-FTg6a7lL7dxXt4oa .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-FTg6a7lL7dxXt4oa .rough-node .label text,#mermaid-svg-FTg6a7lL7dxXt4oa .node .label text,#mermaid-svg-FTg6a7lL7dxXt4oa .image-shape .label,#mermaid-svg-FTg6a7lL7dxXt4oa .icon-shape .label{text-anchor:middle;}#mermaid-svg-FTg6a7lL7dxXt4oa .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-FTg6a7lL7dxXt4oa .rough-node .label,#mermaid-svg-FTg6a7lL7dxXt4oa .node .label,#mermaid-svg-FTg6a7lL7dxXt4oa .image-shape .label,#mermaid-svg-FTg6a7lL7dxXt4oa .icon-shape .label{text-align:center;}#mermaid-svg-FTg6a7lL7dxXt4oa .node.clickable{cursor:pointer;}#mermaid-svg-FTg6a7lL7dxXt4oa .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-FTg6a7lL7dxXt4oa .arrowheadPath{fill:#333333;}#mermaid-svg-FTg6a7lL7dxXt4oa .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-FTg6a7lL7dxXt4oa .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-FTg6a7lL7dxXt4oa .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FTg6a7lL7dxXt4oa .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-FTg6a7lL7dxXt4oa .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FTg6a7lL7dxXt4oa .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-FTg6a7lL7dxXt4oa .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-FTg6a7lL7dxXt4oa .cluster text{fill:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa .cluster span{color:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-FTg6a7lL7dxXt4oa .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-FTg6a7lL7dxXt4oa rect.text{fill:none;stroke-width:0;}#mermaid-svg-FTg6a7lL7dxXt4oa .icon-shape,#mermaid-svg-FTg6a7lL7dxXt4oa .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FTg6a7lL7dxXt4oa .icon-shape p,#mermaid-svg-FTg6a7lL7dxXt4oa .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-FTg6a7lL7dxXt4oa .icon-shape .label rect,#mermaid-svg-FTg6a7lL7dxXt4oa .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FTg6a7lL7dxXt4oa .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-FTg6a7lL7dxXt4oa .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-FTg6a7lL7dxXt4oa :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

1. 加载模型参数 ~14GB (7B FP16)

2. 加载 prompt embeddings n × d × 2B

3. 计算中间激活 (大!) O(n × d × L)

4. 写入 KV Cache 2 × L × n × d × h × 2B

中间激活的显存占用是 Prefill 的关键瓶颈：

存储所有层的中间激活 ≈ L × (n × d + n × d_ff + n × d) × 2 bytes (FP16)

以 LLaMA-7B（L=32, d=4096, d_ff=11008）为例，prompt=4096 时：

• 中间激活 ≈ 32 × (4096×4096 + 4096×11008 + 4096×4096) × 2 ≈ 5.0 GB
• KV Cache ≈ 2 × 32 × 4096 × 4096 × 2 B × (1 head) = 2.1 GB（MQA）

2.5 Prefill 的优化哲学

🌊 3. 幕落 · Decode 阶段深度解构

3.1 定义与边界

Decode 阶段是自回归生成的核心循环。每一步只输入前一步的输出 token（一个 token），借助 KV Cache 计算 attention，生成下一个 token 的 logits。

#mermaid-svg-A9kqkFYet72My6Bp{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-A9kqkFYet72My6Bp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-A9kqkFYet72My6Bp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-A9kqkFYet72My6Bp .error-icon{fill:#552222;}#mermaid-svg-A9kqkFYet72My6Bp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-A9kqkFYet72My6Bp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-A9kqkFYet72My6Bp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-A9kqkFYet72My6Bp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-A9kqkFYet72My6Bp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-A9kqkFYet72My6Bp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-A9kqkFYet72My6Bp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-A9kqkFYet72My6Bp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-A9kqkFYet72My6Bp .marker.cross{stroke:#333333;}#mermaid-svg-A9kqkFYet72My6Bp svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-A9kqkFYet72My6Bp p{margin:0;}#mermaid-svg-A9kqkFYet72My6Bp .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-A9kqkFYet72My6Bp .cluster-label text{fill:#333;}#mermaid-svg-A9kqkFYet72My6Bp .cluster-label span{color:#333;}#mermaid-svg-A9kqkFYet72My6Bp .cluster-label span p{background-color:transparent;}#mermaid-svg-A9kqkFYet72My6Bp .label text,#mermaid-svg-A9kqkFYet72My6Bp span{fill:#333;color:#333;}#mermaid-svg-A9kqkFYet72My6Bp .node rect,#mermaid-svg-A9kqkFYet72My6Bp .node circle,#mermaid-svg-A9kqkFYet72My6Bp .node ellipse,#mermaid-svg-A9kqkFYet72My6Bp .node polygon,#mermaid-svg-A9kqkFYet72My6Bp .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-A9kqkFYet72My6Bp .rough-node .label text,#mermaid-svg-A9kqkFYet72My6Bp .node .label text,#mermaid-svg-A9kqkFYet72My6Bp .image-shape .label,#mermaid-svg-A9kqkFYet72My6Bp .icon-shape .label{text-anchor:middle;}#mermaid-svg-A9kqkFYet72My6Bp .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-A9kqkFYet72My6Bp .rough-node .label,#mermaid-svg-A9kqkFYet72My6Bp .node .label,#mermaid-svg-A9kqkFYet72My6Bp .image-shape .label,#mermaid-svg-A9kqkFYet72My6Bp .icon-shape .label{text-align:center;}#mermaid-svg-A9kqkFYet72My6Bp .node.clickable{cursor:pointer;}#mermaid-svg-A9kqkFYet72My6Bp .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-A9kqkFYet72My6Bp .arrowheadPath{fill:#333333;}#mermaid-svg-A9kqkFYet72My6Bp .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-A9kqkFYet72My6Bp .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-A9kqkFYet72My6Bp .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-A9kqkFYet72My6Bp .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-A9kqkFYet72My6Bp .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-A9kqkFYet72My6Bp .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-A9kqkFYet72My6Bp .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-A9kqkFYet72My6Bp .cluster text{fill:#333;}#mermaid-svg-A9kqkFYet72My6Bp .cluster span{color:#333;}#mermaid-svg-A9kqkFYet72My6Bp div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-A9kqkFYet72My6Bp .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-A9kqkFYet72My6Bp rect.text{fill:none;stroke-width:0;}#mermaid-svg-A9kqkFYet72My6Bp .icon-shape,#mermaid-svg-A9kqkFYet72My6Bp .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-A9kqkFYet72My6Bp .icon-shape p,#mermaid-svg-A9kqkFYet72My6Bp .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-A9kqkFYet72My6Bp .icon-shape .label rect,#mermaid-svg-A9kqkFYet72My6Bp .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-A9kqkFYet72My6Bp .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-A9kqkFYet72My6Bp .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-A9kqkFYet72My6Bp :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Decode 第 t 步

读取

写入

KV Cache (已有 t-1 个 token)

Layer 1: Kʳ, Vʳ

Layer L: Kʳ, Vʳ

当前 Token y_{t-1}

Token Embedding

Q 投影 Qₜ = xₜW_Q

Qₜ · K_cacheᵀ

Softmax

Aₜ · V_cache

追加 Kₜ, Vₜ 到 Cache

注：图中的 r 表示 ℝ⁽ᵗ⁻¹⁾ˣᵈ

3.2 数学形式化（与 Prefill 对比）

关键注意力行向量计算：

Qₜ = xₜ · W_Q # ℝ¹ˣᵈ

# 读取缓存
K_cache = [K₁; K₂; …; K_{t-1}] # ℝ⁽ᵗ⁻¹⁾ˣᵈ
V_cache = [V₁; V₂; …; V_{t-1}] # ℝ⁽ᵗ⁻¹⁾ˣᵈ

# 注意力
Sₜ = Qₜ · K_cacheᵀ # ℝ¹ˣ⁽ᵗ⁻¹⁾
Aₜ = softmax(Sₜ / √dₖ) # ℝ¹ˣ⁽ᵗ⁻¹⁾
Oₜ = Aₜ · V_cache # ℝ¹ˣᵈ

# 追加当前 token 的 KV
K_new = xₜ · W_K # ℝ¹ˣᵈ
V_new = xₜ · W_V # ℝ¹ˣᵈ
K_cache = [K_cache; K_new] # ℝᵗˣᵈ
V_cache = [V_cache; V_new] # ℝᵗˣᵈ

3.3 Decode 的运算特征

Decode 核心: Qₜ(1×d) × K_cacheᵀ(d×(t-1)) = Score(1×(t-1))
Prefill 核心: Q(n×d) × Kᵀ(d×n) = Score(n×n)

对比:
Prefill → GEMM (矩阵×矩阵) → 计算密度高
Decode → GEMV (矩阵×向量) → 计算密度低

GEMV (矩阵-向量乘) 是 Decode 效率低下的根源：

Decode 的每步计算量很少（~2t·d FLOPs，其中 t 是当前生成长度），但需要从 HBM 读取 整个 KV Cache（~2t·d·L·h bytes），计算密度极低，被内存带宽死死卡住。

3.4 Decode 阶段的内存流

#mermaid-svg-2imnaXOYweJTz2hq{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2imnaXOYweJTz2hq .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2imnaXOYweJTz2hq .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2imnaXOYweJTz2hq .error-icon{fill:#552222;}#mermaid-svg-2imnaXOYweJTz2hq .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2imnaXOYweJTz2hq .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2imnaXOYweJTz2hq .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2imnaXOYweJTz2hq .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2imnaXOYweJTz2hq .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2imnaXOYweJTz2hq .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2imnaXOYweJTz2hq .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2imnaXOYweJTz2hq .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2imnaXOYweJTz2hq .marker.cross{stroke:#333333;}#mermaid-svg-2imnaXOYweJTz2hq svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2imnaXOYweJTz2hq p{margin:0;}#mermaid-svg-2imnaXOYweJTz2hq .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-2imnaXOYweJTz2hq .cluster-label text{fill:#333;}#mermaid-svg-2imnaXOYweJTz2hq .cluster-label span{color:#333;}#mermaid-svg-2imnaXOYweJTz2hq .cluster-label span p{background-color:transparent;}#mermaid-svg-2imnaXOYweJTz2hq .label text,#mermaid-svg-2imnaXOYweJTz2hq span{fill:#333;color:#333;}#mermaid-svg-2imnaXOYweJTz2hq .node rect,#mermaid-svg-2imnaXOYweJTz2hq .node circle,#mermaid-svg-2imnaXOYweJTz2hq .node ellipse,#mermaid-svg-2imnaXOYweJTz2hq .node polygon,#mermaid-svg-2imnaXOYweJTz2hq .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2imnaXOYweJTz2hq .rough-node .label text,#mermaid-svg-2imnaXOYweJTz2hq .node .label text,#mermaid-svg-2imnaXOYweJTz2hq .image-shape .label,#mermaid-svg-2imnaXOYweJTz2hq .icon-shape .label{text-anchor:middle;}#mermaid-svg-2imnaXOYweJTz2hq .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2imnaXOYweJTz2hq .rough-node .label,#mermaid-svg-2imnaXOYweJTz2hq .node .label,#mermaid-svg-2imnaXOYweJTz2hq .image-shape .label,#mermaid-svg-2imnaXOYweJTz2hq .icon-shape .label{text-align:center;}#mermaid-svg-2imnaXOYweJTz2hq .node.clickable{cursor:pointer;}#mermaid-svg-2imnaXOYweJTz2hq .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2imnaXOYweJTz2hq .arrowheadPath{fill:#333333;}#mermaid-svg-2imnaXOYweJTz2hq .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2imnaXOYweJTz2hq .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2imnaXOYweJTz2hq .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2imnaXOYweJTz2hq .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2imnaXOYweJTz2hq .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2imnaXOYweJTz2hq .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2imnaXOYweJTz2hq .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2imnaXOYweJTz2hq .cluster text{fill:#333;}#mermaid-svg-2imnaXOYweJTz2hq .cluster span{color:#333;}#mermaid-svg-2imnaXOYweJTz2hq div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2imnaXOYweJTz2hq .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2imnaXOYweJTz2hq rect.text{fill:none;stroke-width:0;}#mermaid-svg-2imnaXOYweJTz2hq .icon-shape,#mermaid-svg-2imnaXOYweJTz2hq .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2imnaXOYweJTz2hq .icon-shape p,#mermaid-svg-2imnaXOYweJTz2hq .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2imnaXOYweJTz2hq .icon-shape .label rect,#mermaid-svg-2imnaXOYweJTz2hq .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2imnaXOYweJTz2hq .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2imnaXOYweJTz2hq .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2imnaXOYweJTz2hq :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

读取 Model Weights ~14GB

Attention Q 投影

读取 KV Cache 2×L×t×d×h×2B

Score = Q · K_cacheᵀ

A · V_cache

写入新 KV 2×L×d×2B

💡 主要瓶颈：KV Cache 读取带宽，随 t 增长而增长

3.5 Decode 延迟拆解

以 LLaMA-7B, FP16, 单卡 A100 (80GB, 2TB/s HBM) 为例：

KV Cache 读取占总延迟的 ~60%，是 Decode 优化的核心目标。

⚖️ 4. 横切对比：两张面孔

4.1 全景对比表

🔥 Prefill ❄️ Decode
===== =====
并行处理 N 个 token 串行逐 token 生成
GEMM 计算密集 GEMV 带宽密集
GPU 利用率 70-90% GPU 利用率 5-20%
建立 KV Cache 复用 + 追加 KV Cache
延迟 O(n²) 延迟 O(n) per step
主要瓶颈：计算 主要瓶颈：带宽

4.2 Attention 形状可视化

Prefill Attention: Decode 第 t 步:
n×n 矩阵 1×(t-1) 行向量

Q₁ Q₂ Q₃ Q₄ Qₜ
K₁ [ · · · · ] K₁ [ · ]
K₂ [ · · · · ] → K₂ [ · ]
K₃ [ · · · · ] K₃ [ · ]
K₄ [ · · · · ] K₄ [ · ]
…

Prefill 的 attention 矩阵是 n×n（每行对应一个 query 位置），Decode 的 attention 是 1×(t-1) 的行向量。

4.3 计算图对比

#mermaid-svg-0w2hCvfoDtdb8PL3{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-0w2hCvfoDtdb8PL3 .error-icon{fill:#552222;}#mermaid-svg-0w2hCvfoDtdb8PL3 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-0w2hCvfoDtdb8PL3 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .marker.cross{stroke:#333333;}#mermaid-svg-0w2hCvfoDtdb8PL3 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-0w2hCvfoDtdb8PL3 p{margin:0;}#mermaid-svg-0w2hCvfoDtdb8PL3 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .cluster-label text{fill:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .cluster-label span{color:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .cluster-label span p{background-color:transparent;}#mermaid-svg-0w2hCvfoDtdb8PL3 .label text,#mermaid-svg-0w2hCvfoDtdb8PL3 span{fill:#333;color:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .node rect,#mermaid-svg-0w2hCvfoDtdb8PL3 .node circle,#mermaid-svg-0w2hCvfoDtdb8PL3 .node ellipse,#mermaid-svg-0w2hCvfoDtdb8PL3 .node polygon,#mermaid-svg-0w2hCvfoDtdb8PL3 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .rough-node .label text,#mermaid-svg-0w2hCvfoDtdb8PL3 .node .label text,#mermaid-svg-0w2hCvfoDtdb8PL3 .image-shape .label,#mermaid-svg-0w2hCvfoDtdb8PL3 .icon-shape .label{text-anchor:middle;}#mermaid-svg-0w2hCvfoDtdb8PL3 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .rough-node .label,#mermaid-svg-0w2hCvfoDtdb8PL3 .node .label,#mermaid-svg-0w2hCvfoDtdb8PL3 .image-shape .label,#mermaid-svg-0w2hCvfoDtdb8PL3 .icon-shape .label{text-align:center;}#mermaid-svg-0w2hCvfoDtdb8PL3 .node.clickable{cursor:pointer;}#mermaid-svg-0w2hCvfoDtdb8PL3 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .arrowheadPath{fill:#333333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0w2hCvfoDtdb8PL3 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-0w2hCvfoDtdb8PL3 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0w2hCvfoDtdb8PL3 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-0w2hCvfoDtdb8PL3 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .cluster text{fill:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 .cluster span{color:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-0w2hCvfoDtdb8PL3 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-0w2hCvfoDtdb8PL3 rect.text{fill:none;stroke-width:0;}#mermaid-svg-0w2hCvfoDtdb8PL3 .icon-shape,#mermaid-svg-0w2hCvfoDtdb8PL3 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0w2hCvfoDtdb8PL3 .icon-shape p,#mermaid-svg-0w2hCvfoDtdb8PL3 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-0w2hCvfoDtdb8PL3 .icon-shape .label rect,#mermaid-svg-0w2hCvfoDtdb8PL3 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0w2hCvfoDtdb8PL3 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-0w2hCvfoDtdb8PL3 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-0w2hCvfoDtdb8PL3 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Decode 计算图

Qₜ (1×d)

Score (1×(t-1))

Probs (1×(t-1))

Out (1×d)

K_cache ((t-1)×d)

V_cache ((t-1)×d)

Prefill 计算图

Q (n×d)

Score (n×n)

Probs (n×n)

Out (n×d)

K (n×d)

V (n×d)

核心区别：Prefill 的 Score 是 GEMM，Decode 的 Score 是 GEMV。

🧩 5. KV Cache 微观解剖

5.1 数据结构与形状

KV Cache 从本质上说，就是各层各 head 的 Key 和 Value 矩阵的逐行追加缓存：

KV Cache 形状: [L, 2, h, t, dₖ]
↑ ↑ ↑ ↑ ↑
层 K/V 头 序列 每头维度

#mermaid-svg-ozq903EluF7QPZHp{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ozq903EluF7QPZHp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ozq903EluF7QPZHp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ozq903EluF7QPZHp .error-icon{fill:#552222;}#mermaid-svg-ozq903EluF7QPZHp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ozq903EluF7QPZHp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ozq903EluF7QPZHp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ozq903EluF7QPZHp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ozq903EluF7QPZHp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ozq903EluF7QPZHp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ozq903EluF7QPZHp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ozq903EluF7QPZHp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ozq903EluF7QPZHp .marker.cross{stroke:#333333;}#mermaid-svg-ozq903EluF7QPZHp svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ozq903EluF7QPZHp p{margin:0;}#mermaid-svg-ozq903EluF7QPZHp .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-ozq903EluF7QPZHp .cluster-label text{fill:#333;}#mermaid-svg-ozq903EluF7QPZHp .cluster-label span{color:#333;}#mermaid-svg-ozq903EluF7QPZHp .cluster-label span p{background-color:transparent;}#mermaid-svg-ozq903EluF7QPZHp .label text,#mermaid-svg-ozq903EluF7QPZHp span{fill:#333;color:#333;}#mermaid-svg-ozq903EluF7QPZHp .node rect,#mermaid-svg-ozq903EluF7QPZHp .node circle,#mermaid-svg-ozq903EluF7QPZHp .node ellipse,#mermaid-svg-ozq903EluF7QPZHp .node polygon,#mermaid-svg-ozq903EluF7QPZHp .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ozq903EluF7QPZHp .rough-node .label text,#mermaid-svg-ozq903EluF7QPZHp .node .label text,#mermaid-svg-ozq903EluF7QPZHp .image-shape .label,#mermaid-svg-ozq903EluF7QPZHp .icon-shape .label{text-anchor:middle;}#mermaid-svg-ozq903EluF7QPZHp .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ozq903EluF7QPZHp .rough-node .label,#mermaid-svg-ozq903EluF7QPZHp .node .label,#mermaid-svg-ozq903EluF7QPZHp .image-shape .label,#mermaid-svg-ozq903EluF7QPZHp .icon-shape .label{text-align:center;}#mermaid-svg-ozq903EluF7QPZHp .node.clickable{cursor:pointer;}#mermaid-svg-ozq903EluF7QPZHp .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ozq903EluF7QPZHp .arrowheadPath{fill:#333333;}#mermaid-svg-ozq903EluF7QPZHp .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ozq903EluF7QPZHp .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ozq903EluF7QPZHp .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ozq903EluF7QPZHp .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ozq903EluF7QPZHp .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ozq903EluF7QPZHp .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ozq903EluF7QPZHp .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ozq903EluF7QPZHp .cluster text{fill:#333;}#mermaid-svg-ozq903EluF7QPZHp .cluster span{color:#333;}#mermaid-svg-ozq903EluF7QPZHp div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ozq903EluF7QPZHp .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ozq903EluF7QPZHp rect.text{fill:none;stroke-width:0;}#mermaid-svg-ozq903EluF7QPZHp .icon-shape,#mermaid-svg-ozq903EluF7QPZHp .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ozq903EluF7QPZHp .icon-shape p,#mermaid-svg-ozq903EluF7QPZHp .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ozq903EluF7QPZHp .icon-shape .label rect,#mermaid-svg-ozq903EluF7QPZHp .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ozq903EluF7QPZHp .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ozq903EluF7QPZHp .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ozq903EluF7QPZHp :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

整体形状: [L, 2, max_seq_len, num_kv_heads, head_dim]

— Layer 0 —

K: [num_kv_heads, max_seq_len, head_dim]

V: [num_kv_heads, max_seq_len, head_dim]

— Layer 1 —

K: [num_kv_heads, max_seq_len, head_dim]

V: [num_kv_heads, max_seq_len, head_dim]

— Layer L-1 —

K: [num_kv_heads, max_seq_len, head_dim]

V: [num_kv_heads, max_seq_len, head_dim]

5.2 不同 Attention 变体的 KV Cache 大小

计算公式：

KV Cache 大小 = 2 (K+V) × L × t × g × dₖ × sizeof(dtype)

5.3 KV Cache 的内存布局

不同的推理引擎对 KV Cache 的内存布局有不同的选择：

#mermaid-svg-Cp6wuqmwUy8Xg9TK{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .error-icon{fill:#552222;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .marker.cross{stroke:#333333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK p{margin:0;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .cluster-label text{fill:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .cluster-label span{color:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .cluster-label span p{background-color:transparent;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .label text,#mermaid-svg-Cp6wuqmwUy8Xg9TK span{fill:#333;color:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .node rect,#mermaid-svg-Cp6wuqmwUy8Xg9TK .node circle,#mermaid-svg-Cp6wuqmwUy8Xg9TK .node ellipse,#mermaid-svg-Cp6wuqmwUy8Xg9TK .node polygon,#mermaid-svg-Cp6wuqmwUy8Xg9TK .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .rough-node .label text,#mermaid-svg-Cp6wuqmwUy8Xg9TK .node .label text,#mermaid-svg-Cp6wuqmwUy8Xg9TK .image-shape .label,#mermaid-svg-Cp6wuqmwUy8Xg9TK .icon-shape .label{text-anchor:middle;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .rough-node .label,#mermaid-svg-Cp6wuqmwUy8Xg9TK .node .label,#mermaid-svg-Cp6wuqmwUy8Xg9TK .image-shape .label,#mermaid-svg-Cp6wuqmwUy8Xg9TK .icon-shape .label{text-align:center;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .node.clickable{cursor:pointer;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .arrowheadPath{fill:#333333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Cp6wuqmwUy8Xg9TK .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Cp6wuqmwUy8Xg9TK .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Cp6wuqmwUy8Xg9TK .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .cluster text{fill:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .cluster span{color:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Cp6wuqmwUy8Xg9TK rect.text{fill:none;stroke-width:0;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .icon-shape,#mermaid-svg-Cp6wuqmwUy8Xg9TK .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .icon-shape p,#mermaid-svg-Cp6wuqmwUy8Xg9TK .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .icon-shape .label rect,#mermaid-svg-Cp6wuqmwUy8Xg9TK .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Cp6wuqmwUy8Xg9TK .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Cp6wuqmwUy8Xg9TK .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Cp6wuqmwUy8Xg9TK :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Contiguous (连续布局) K: [L, h, max_len, dₖ] 连续存储，简单但碎片多 TensorRT-LLM

Paged (分页布局) 按 Block 管理 (16/32 token/block) 类 OS 虚拟内存，零碎片 vLLM / LightLLM

Flat (扁平布局) 拼接所有 (K, V) 对为连续 buffer 配合 PageAttention 使用 vLLM 内部实现

分配一整块连续显存，浪费 padding 空间

按需分配 block，浪费 block 内最后一个 token

一维大数组 + 索引表，浪费预留的 max_len 空间

5.4 KV Cache 的增长过程

Decode 步

KV Cache

Decode 步

KV Cache

#mermaid-svg-BEnUOWKCcf90k1j3{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-BEnUOWKCcf90k1j3 .error-icon{fill:#552222;}#mermaid-svg-BEnUOWKCcf90k1j3 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-BEnUOWKCcf90k1j3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-BEnUOWKCcf90k1j3 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-BEnUOWKCcf90k1j3 .marker.cross{stroke:#333333;}#mermaid-svg-BEnUOWKCcf90k1j3 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-BEnUOWKCcf90k1j3 p{margin:0;}#mermaid-svg-BEnUOWKCcf90k1j3 .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-BEnUOWKCcf90k1j3 text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-BEnUOWKCcf90k1j3 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-BEnUOWKCcf90k1j3 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-BEnUOWKCcf90k1j3 #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-BEnUOWKCcf90k1j3 .sequenceNumber{fill:white;}#mermaid-svg-BEnUOWKCcf90k1j3 #sequencenumber{fill:#333;}#mermaid-svg-BEnUOWKCcf90k1j3 #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-BEnUOWKCcf90k1j3 .messageText{fill:#333;stroke:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-BEnUOWKCcf90k1j3 .labelText,#mermaid-svg-BEnUOWKCcf90k1j3 .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .loopText,#mermaid-svg-BEnUOWKCcf90k1j3 .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-BEnUOWKCcf90k1j3 .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-BEnUOWKCcf90k1j3 .noteText,#mermaid-svg-BEnUOWKCcf90k1j3 .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-BEnUOWKCcf90k1j3 .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-BEnUOWKCcf90k1j3 .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-BEnUOWKCcf90k1j3 .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-BEnUOWKCcf90k1j3 .actorPopupMenu{position:absolute;}#mermaid-svg-BEnUOWKCcf90k1j3 .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-BEnUOWKCcf90k1j3 .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-BEnUOWKCcf90k1j3 .actor-man circle,#mermaid-svg-BEnUOWKCcf90k1j3 line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-BEnUOWKCcf90k1j3 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Prefill 后初始化, t = n (prompt 长度)

计算 Attention 输出

t += 1

loop

[每步 Decode]

直到 EOS 或 max_tokens

读取 K_cache[0:t], V_cache[0:t]

追加 K_new, V_new

返回更新后的 Cache

5.5 KV Cache 量化

KV Cache 是 Decode 阶段最大的内存和带宽消费者，量化是必然选择：

#mermaid-svg-kwojOrmt9MkJB6TY{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-kwojOrmt9MkJB6TY .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-kwojOrmt9MkJB6TY .error-icon{fill:#552222;}#mermaid-svg-kwojOrmt9MkJB6TY .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-kwojOrmt9MkJB6TY .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-kwojOrmt9MkJB6TY .marker{fill:#333333;stroke:#333333;}#mermaid-svg-kwojOrmt9MkJB6TY .marker.cross{stroke:#333333;}#mermaid-svg-kwojOrmt9MkJB6TY svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-kwojOrmt9MkJB6TY p{margin:0;}#mermaid-svg-kwojOrmt9MkJB6TY .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-kwojOrmt9MkJB6TY .cluster-label text{fill:#333;}#mermaid-svg-kwojOrmt9MkJB6TY .cluster-label span{color:#333;}#mermaid-svg-kwojOrmt9MkJB6TY .cluster-label span p{background-color:transparent;}#mermaid-svg-kwojOrmt9MkJB6TY .label text,#mermaid-svg-kwojOrmt9MkJB6TY span{fill:#333;color:#333;}#mermaid-svg-kwojOrmt9MkJB6TY .node rect,#mermaid-svg-kwojOrmt9MkJB6TY .node circle,#mermaid-svg-kwojOrmt9MkJB6TY .node ellipse,#mermaid-svg-kwojOrmt9MkJB6TY .node polygon,#mermaid-svg-kwojOrmt9MkJB6TY .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-kwojOrmt9MkJB6TY .rough-node .label text,#mermaid-svg-kwojOrmt9MkJB6TY .node .label text,#mermaid-svg-kwojOrmt9MkJB6TY .image-shape .label,#mermaid-svg-kwojOrmt9MkJB6TY .icon-shape .label{text-anchor:middle;}#mermaid-svg-kwojOrmt9MkJB6TY .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-kwojOrmt9MkJB6TY .rough-node .label,#mermaid-svg-kwojOrmt9MkJB6TY .node .label,#mermaid-svg-kwojOrmt9MkJB6TY .image-shape .label,#mermaid-svg-kwojOrmt9MkJB6TY .icon-shape .label{text-align:center;}#mermaid-svg-kwojOrmt9MkJB6TY .node.clickable{cursor:pointer;}#mermaid-svg-kwojOrmt9MkJB6TY .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-kwojOrmt9MkJB6TY .arrowheadPath{fill:#333333;}#mermaid-svg-kwojOrmt9MkJB6TY .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-kwojOrmt9MkJB6TY .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-kwojOrmt9MkJB6TY .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-kwojOrmt9MkJB6TY .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-kwojOrmt9MkJB6TY .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-kwojOrmt9MkJB6TY .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-kwojOrmt9MkJB6TY .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-kwojOrmt9MkJB6TY .cluster text{fill:#333;}#mermaid-svg-kwojOrmt9MkJB6TY .cluster span{color:#333;}#mermaid-svg-kwojOrmt9MkJB6TY div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-kwojOrmt9MkJB6TY .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-kwojOrmt9MkJB6TY rect.text{fill:none;stroke-width:0;}#mermaid-svg-kwojOrmt9MkJB6TY .icon-shape,#mermaid-svg-kwojOrmt9MkJB6TY .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-kwojOrmt9MkJB6TY .icon-shape p,#mermaid-svg-kwojOrmt9MkJB6TY .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-kwojOrmt9MkJB6TY .icon-shape .label rect,#mermaid-svg-kwojOrmt9MkJB6TY .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-kwojOrmt9MkJB6TY .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-kwojOrmt9MkJB6TY .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-kwojOrmt9MkJB6TY :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

K (FP16)

计算 Scale per-token 或 per-channel

量化到 INT8 K_quant = K / scale

存储 K_quant + scale + zero_point

在线反量化 K_deq = K_quant * scale

Attention 计算

KVCache 量化的关键挑战：

• 量化粒度的选择：per-token, per-channel, per-block
• 反量化的计算开销 vs 减少内存读取的收益
• Outlier 处理（某些 channel 的数值范围极大）

📈 6. 天花板在哪：Roofline 模型分析

6.1 Roofline 基础

Roofline 模型揭示了 Prefill 和 Decode 分别受到什么限制：

A100 Roofline 参数:
峰值计算: 312 TFLOPS (FP16 Tensor Core)
峰值带宽: 1.6 TB/s (HBM2e, 官方) / 2.0 TB/s (实际)
Ridge Point: 312 / 1.6 ≈ 195 FLOPs/Byte

计算受限区（← Prefill 在此）: 计算密度 > 195 FLOPs/Byte
带宽受限区（← Decode 在此）: 计算密度 < 195 FLOPs/Byte

6.2 具体计算

以 LLaMA-7B (d=4096, d_ff=11008, L=32) 为例：

Prefill (n=2048):

QKᵀ: Q (2048×4096) × Kᵀ (4096×2048) = 2 × 2048 × 4096 × 2048 = 34.4 GFLOPs
读取: Q (2048×4096×2=16MB), Kᵀ (4096×2048×2=16MB) = 32MB
计算密度 ≈ 34.4 / 0.032 ≈ 1075 FLOPs/Byte > 195 → 计算受限 ✓

Decode (t=2048):

QKᵀ: Q (1×4096) × Kᵀ (4096×2048) = 2 × 1 × 4096 × 2048 = 16.8 MFLOPs
读取: Q (4096×2=8KB), K (2048×4096×2=16MB) = ~16MB
计算密度 ≈ 16.8M / 16M ≈ 1.05 FLOPs/Byte << 195 → 带宽受限 ✓

Decode 的计算密度比 Ridge Point 低 ~100-200 倍，这就是为什么 Decode 效率如此之低的根本原因。

6.3 算术强度与硬件匹配

🚀 7. 前沿优化技术全景

7.1 优化分类图谱

#mermaid-svg-IAQOR7lzg6CsDXEf{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IAQOR7lzg6CsDXEf .error-icon{fill:#552222;}#mermaid-svg-IAQOR7lzg6CsDXEf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IAQOR7lzg6CsDXEf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IAQOR7lzg6CsDXEf .marker.cross{stroke:#333333;}#mermaid-svg-IAQOR7lzg6CsDXEf svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IAQOR7lzg6CsDXEf p{margin:0;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge{stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 text{fill:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth–1{stroke-width:17;}#mermaid-svg-IAQOR7lzg6CsDXEf .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-0{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-0{stroke-width:14;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-1{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-1{stroke-width:11;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 text{fill:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-2{stroke-width:8;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-3{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-3{stroke-width:5;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-4{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-4{stroke-width:2;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-5{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-5{stroke-width:-1;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-6{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-6{stroke-width:-4;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-7{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-7{stroke-width:-7;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-8{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-8{stroke-width:-10;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-9{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-9{stroke-width:-13;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 polygon,#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 text{fill:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .node-icon-10{font-size:40px;color:black;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .edge-depth-10{stroke-width:-16;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled circle,#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:lightgray;}#mermaid-svg-IAQOR7lzg6CsDXEf .disabled text{fill:#efefef;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-root rect,#mermaid-svg-IAQOR7lzg6CsDXEf .section-root path,#mermaid-svg-IAQOR7lzg6CsDXEf .section-root circle,#mermaid-svg-IAQOR7lzg6CsDXEf .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-IAQOR7lzg6CsDXEf .section-root text{fill:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-root span{color:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .section-2 span{color:#ffffff;}#mermaid-svg-IAQOR7lzg6CsDXEf .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-IAQOR7lzg6CsDXEf .edge{fill:none;}#mermaid-svg-IAQOR7lzg6CsDXEf .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-IAQOR7lzg6CsDXEf :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Decode-Only 优化全景

Prefill 优化

Flash Attention 系列

PageAttention

Continuous Batching

Prefix Caching

Chunked Prefill

Decode 优化

Speculative Decoding

Medusa / Eagle

Lookahead Decoding

Streaming LLM

MQA / GQA

KV Cache 优化

INT8/FP8/INT4

PagedAttention

CPU/SSD

RadixAttention

窗口化 / 剪枝

系统级优化

动态调度

TP/PP/DP

W4A16 / W8A16

TensorRT / CUDA Graph

7.2 Flash Attention：Prefill 的革命

Flash Attention 通过分块计算（tiling）和重计算（recomputation），将 attention 的 HBM 读写从 O(n²) 降低到 O(n)。

#mermaid-svg-n7VNlKQtMBXzNR02{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-n7VNlKQtMBXzNR02 .error-icon{fill:#552222;}#mermaid-svg-n7VNlKQtMBXzNR02 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-n7VNlKQtMBXzNR02 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-n7VNlKQtMBXzNR02 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-n7VNlKQtMBXzNR02 .marker.cross{stroke:#333333;}#mermaid-svg-n7VNlKQtMBXzNR02 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-n7VNlKQtMBXzNR02 p{margin:0;}#mermaid-svg-n7VNlKQtMBXzNR02 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 .cluster-label text{fill:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 .cluster-label span{color:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 .cluster-label span p{background-color:transparent;}#mermaid-svg-n7VNlKQtMBXzNR02 .label text,#mermaid-svg-n7VNlKQtMBXzNR02 span{fill:#333;color:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 .node rect,#mermaid-svg-n7VNlKQtMBXzNR02 .node circle,#mermaid-svg-n7VNlKQtMBXzNR02 .node ellipse,#mermaid-svg-n7VNlKQtMBXzNR02 .node polygon,#mermaid-svg-n7VNlKQtMBXzNR02 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-n7VNlKQtMBXzNR02 .rough-node .label text,#mermaid-svg-n7VNlKQtMBXzNR02 .node .label text,#mermaid-svg-n7VNlKQtMBXzNR02 .image-shape .label,#mermaid-svg-n7VNlKQtMBXzNR02 .icon-shape .label{text-anchor:middle;}#mermaid-svg-n7VNlKQtMBXzNR02 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-n7VNlKQtMBXzNR02 .rough-node .label,#mermaid-svg-n7VNlKQtMBXzNR02 .node .label,#mermaid-svg-n7VNlKQtMBXzNR02 .image-shape .label,#mermaid-svg-n7VNlKQtMBXzNR02 .icon-shape .label{text-align:center;}#mermaid-svg-n7VNlKQtMBXzNR02 .node.clickable{cursor:pointer;}#mermaid-svg-n7VNlKQtMBXzNR02 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-n7VNlKQtMBXzNR02 .arrowheadPath{fill:#333333;}#mermaid-svg-n7VNlKQtMBXzNR02 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-n7VNlKQtMBXzNR02 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-n7VNlKQtMBXzNR02 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-n7VNlKQtMBXzNR02 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-n7VNlKQtMBXzNR02 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-n7VNlKQtMBXzNR02 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-n7VNlKQtMBXzNR02 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-n7VNlKQtMBXzNR02 .cluster text{fill:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 .cluster span{color:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-n7VNlKQtMBXzNR02 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-n7VNlKQtMBXzNR02 rect.text{fill:none;stroke-width:0;}#mermaid-svg-n7VNlKQtMBXzNR02 .icon-shape,#mermaid-svg-n7VNlKQtMBXzNR02 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-n7VNlKQtMBXzNR02 .icon-shape p,#mermaid-svg-n7VNlKQtMBXzNR02 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-n7VNlKQtMBXzNR02 .icon-shape .label rect,#mermaid-svg-n7VNlKQtMBXzNR02 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-n7VNlKQtMBXzNR02 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-n7VNlKQtMBXzNR02 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-n7VNlKQtMBXzNR02 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Flash Attention

Q, K, V 分块加载到 SRAM

在 SRAM 内计算局部 Score

在线 Softmax (tiled 算法)

局部输出累积到 HBM

标准 Attention

Q, K, V 读取到 HBM

Score = QKᵀ 写入 HBM (O(n²))

Softmax 从 HBM 读取 score

O = PV 写入 HBM

Flash Attention v1/v2/v3 关键演进：

7.3 Speculative Decoding：打破 Decode 的串行桎梏

核心思想：用小模型草稿 + 大模型验证，一次前向生成多个 token。

输出

目标模型 (~7B)

草稿模型 (~0.5B)

输出

目标模型 (~7B)

草稿模型 (~0.5B)

#mermaid-svg-B6IOKcohM9gadnDn{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-B6IOKcohM9gadnDn .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-B6IOKcohM9gadnDn .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-B6IOKcohM9gadnDn .error-icon{fill:#552222;}#mermaid-svg-B6IOKcohM9gadnDn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-B6IOKcohM9gadnDn .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-B6IOKcohM9gadnDn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-B6IOKcohM9gadnDn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-B6IOKcohM9gadnDn .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-B6IOKcohM9gadnDn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-B6IOKcohM9gadnDn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-B6IOKcohM9gadnDn .marker{fill:#333333;stroke:#333333;}#mermaid-svg-B6IOKcohM9gadnDn .marker.cross{stroke:#333333;}#mermaid-svg-B6IOKcohM9gadnDn svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-B6IOKcohM9gadnDn p{margin:0;}#mermaid-svg-B6IOKcohM9gadnDn .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-B6IOKcohM9gadnDn text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-B6IOKcohM9gadnDn .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-B6IOKcohM9gadnDn .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-B6IOKcohM9gadnDn .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-B6IOKcohM9gadnDn .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-B6IOKcohM9gadnDn #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-B6IOKcohM9gadnDn .sequenceNumber{fill:white;}#mermaid-svg-B6IOKcohM9gadnDn #sequencenumber{fill:#333;}#mermaid-svg-B6IOKcohM9gadnDn #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-B6IOKcohM9gadnDn .messageText{fill:#333;stroke:none;}#mermaid-svg-B6IOKcohM9gadnDn .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-B6IOKcohM9gadnDn .labelText,#mermaid-svg-B6IOKcohM9gadnDn .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-B6IOKcohM9gadnDn .loopText,#mermaid-svg-B6IOKcohM9gadnDn .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-B6IOKcohM9gadnDn .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-B6IOKcohM9gadnDn .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-B6IOKcohM9gadnDn .noteText,#mermaid-svg-B6IOKcohM9gadnDn .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-B6IOKcohM9gadnDn .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-B6IOKcohM9gadnDn .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-B6IOKcohM9gadnDn .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-B6IOKcohM9gadnDn .actorPopupMenu{position:absolute;}#mermaid-svg-B6IOKcohM9gadnDn .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-B6IOKcohM9gadnDn .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-B6IOKcohM9gadnDn .actor-man circle,#mermaid-svg-B6IOKcohM9gadnDn line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-B6IOKcohM9gadnDn :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

并行计算 O(gamma) 而非串行 O(gamma)

alt

[全部接受]

[部分接受 (k 个)]

loop

[每轮推测]

自回归生成 gamma 个草稿 token

提交草稿 [y1, y2, …, y_gamma]

Prefill 模式, 一次性验证全部 gamma 个草稿

输出验证结果

产出 gamma 个 token, 效率 = gamma 倍

产出 k 个 token, 丢弃第 k+1 个及后续

效率分析：

加速比 = gamma × (p_accept + p_accept² + … + p_accept^gamma)
× (T_target + T_draft_per_step) / T_target

其中 p_accept 是每个位置草稿被接受的概率
对于理想的 80% 接受率，gamma=5:
– 每轮期望产出 ≈ 3.0 tokens
– 加速比 ≈ 2.5×

7.4 连续批处理 (Continuous Batching)

传统批处理 — 大量气泡：

#mermaid-svg-mcuk29oSRn3YgGFo{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-mcuk29oSRn3YgGFo .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-mcuk29oSRn3YgGFo .error-icon{fill:#552222;}#mermaid-svg-mcuk29oSRn3YgGFo .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-mcuk29oSRn3YgGFo .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-mcuk29oSRn3YgGFo .marker{fill:#333333;stroke:#333333;}#mermaid-svg-mcuk29oSRn3YgGFo .marker.cross{stroke:#333333;}#mermaid-svg-mcuk29oSRn3YgGFo svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-mcuk29oSRn3YgGFo p{margin:0;}#mermaid-svg-mcuk29oSRn3YgGFo .mermaid-main-font{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-mcuk29oSRn3YgGFo .exclude-range{fill:#eeeeee;}#mermaid-svg-mcuk29oSRn3YgGFo .section{stroke:none;opacity:0.2;}#mermaid-svg-mcuk29oSRn3YgGFo .section0{fill:rgba(102, 102, 255, 0.49);}#mermaid-svg-mcuk29oSRn3YgGFo .section2{fill:#fff400;}#mermaid-svg-mcuk29oSRn3YgGFo .section1,#mermaid-svg-mcuk29oSRn3YgGFo .section3{fill:white;opacity:0.2;}#mermaid-svg-mcuk29oSRn3YgGFo .sectionTitle0{fill:#333;}#mermaid-svg-mcuk29oSRn3YgGFo .sectionTitle1{fill:#333;}#mermaid-svg-mcuk29oSRn3YgGFo .sectionTitle2{fill:#333;}#mermaid-svg-mcuk29oSRn3YgGFo .sectionTitle3{fill:#333;}#mermaid-svg-mcuk29oSRn3YgGFo .sectionTitle{text-anchor:start;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-mcuk29oSRn3YgGFo .grid .tick{stroke:lightgrey;opacity:0.8;shape-rendering:crispEdges;}#mermaid-svg-mcuk29oSRn3YgGFo .grid .tick text{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;fill:#333;}#mermaid-svg-mcuk29oSRn3YgGFo .grid path{stroke-width:0;}#mermaid-svg-mcuk29oSRn3YgGFo .today{fill:none;stroke:red;stroke-width:2px;}#mermaid-svg-mcuk29oSRn3YgGFo .task{stroke-width:2;}#mermaid-svg-mcuk29oSRn3YgGFo .taskText{text-anchor:middle;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutsideRight{fill:black;text-anchor:start;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutsideLeft{fill:black;text-anchor:end;}#mermaid-svg-mcuk29oSRn3YgGFo .task.clickable{cursor:pointer;}#mermaid-svg-mcuk29oSRn3YgGFo .taskText.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-mcuk29oSRn3YgGFo .taskText0,#mermaid-svg-mcuk29oSRn3YgGFo .taskText1,#mermaid-svg-mcuk29oSRn3YgGFo .taskText2,#mermaid-svg-mcuk29oSRn3YgGFo .taskText3{fill:white;}#mermaid-svg-mcuk29oSRn3YgGFo .task0,#mermaid-svg-mcuk29oSRn3YgGFo .task1,#mermaid-svg-mcuk29oSRn3YgGFo .task2,#mermaid-svg-mcuk29oSRn3YgGFo .task3{fill:#8a90dd;stroke:#534fbc;}#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutside0,#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutside2{fill:black;}#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutside1,#mermaid-svg-mcuk29oSRn3YgGFo .taskTextOutside3{fill:black;}#mermaid-svg-mcuk29oSRn3YgGFo .active0,#mermaid-svg-mcuk29oSRn3YgGFo .active1,#mermaid-svg-mcuk29oSRn3YgGFo .active2,#mermaid-svg-mcuk29oSRn3YgGFo .active3{fill:#bfc7ff;stroke:#534fbc;}#mermaid-svg-mcuk29oSRn3YgGFo .activeText0,#mermaid-svg-mcuk29oSRn3YgGFo .activeText1,#mermaid-svg-mcuk29oSRn3YgGFo .activeText2,#mermaid-svg-mcuk29oSRn3YgGFo .activeText3{fill:black!important;}#mermaid-svg-mcuk29oSRn3YgGFo .done0,#mermaid-svg-mcuk29oSRn3YgGFo .done1,#mermaid-svg-mcuk29oSRn3YgGFo .done2,#mermaid-svg-mcuk29oSRn3YgGFo .done3{stroke:grey;fill:lightgrey;stroke-width:2;}#mermaid-svg-mcuk29oSRn3YgGFo .doneText0,#mermaid-svg-mcuk29oSRn3YgGFo .doneText1,#mermaid-svg-mcuk29oSRn3YgGFo .doneText2,#mermaid-svg-mcuk29oSRn3YgGFo .doneText3{fill:black!important;}#mermaid-svg-mcuk29oSRn3YgGFo .doneText0.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneText0.taskTextOutsideRight,#mermaid-svg-mcuk29oSRn3YgGFo .doneText1.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneText1.taskTextOutsideRight,#mermaid-svg-mcuk29oSRn3YgGFo .doneText2.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneText2.taskTextOutsideRight,#mermaid-svg-mcuk29oSRn3YgGFo .doneText3.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-mcuk29oSRn3YgGFo .crit0,#mermaid-svg-mcuk29oSRn3YgGFo .crit1,#mermaid-svg-mcuk29oSRn3YgGFo .crit2,#mermaid-svg-mcuk29oSRn3YgGFo .crit3{stroke:#ff8888;fill:red;stroke-width:2;}#mermaid-svg-mcuk29oSRn3YgGFo .activeCrit0,#mermaid-svg-mcuk29oSRn3YgGFo .activeCrit1,#mermaid-svg-mcuk29oSRn3YgGFo .activeCrit2,#mermaid-svg-mcuk29oSRn3YgGFo .activeCrit3{stroke:#ff8888;fill:#bfc7ff;stroke-width:2;}#mermaid-svg-mcuk29oSRn3YgGFo .doneCrit0,#mermaid-svg-mcuk29oSRn3YgGFo .doneCrit1,#mermaid-svg-mcuk29oSRn3YgGFo .doneCrit2,#mermaid-svg-mcuk29oSRn3YgGFo .doneCrit3{stroke:#ff8888;fill:lightgrey;stroke-width:2;cursor:pointer;shape-rendering:crispEdges;}#mermaid-svg-mcuk29oSRn3YgGFo .milestone{transform:rotate(45deg) scale(0.8,0.8);}#mermaid-svg-mcuk29oSRn3YgGFo .milestoneText{font-style:italic;}#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText0,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText1,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText2,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText3{fill:black!important;}#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText0.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText0.taskTextOutsideRight,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText1.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText1.taskTextOutsideRight,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText2.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText2.taskTextOutsideRight,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText3.taskTextOutsideLeft,#mermaid-svg-mcuk29oSRn3YgGFo .doneCritText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-mcuk29oSRn3YgGFo .vert{stroke:navy;}#mermaid-svg-mcuk29oSRn3YgGFo .vertText{font-size:15px;text-anchor:middle;fill:navy!important;}#mermaid-svg-mcuk29oSRn3YgGFo .activeCritText0,#mermaid-svg-mcuk29oSRn3YgGFo .activeCritText1,#mermaid-svg-mcuk29oSRn3YgGFo .activeCritText2,#mermaid-svg-mcuk29oSRn3YgGFo .activeCritText3{fill:black!important;}#mermaid-svg-mcuk29oSRn3YgGFo .titleText{text-anchor:middle;font-size:18px;fill:#333;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-mcuk29oSRn3YgGFo :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

0

1

2

3

4

5

6

7

8

9

10

11

Prefill A

Prefill B

Decode A1

Decode B1

Prefill C

Decode A2

Decode B2

Decode C1

Decode A3

Bubble1

请求 A

请求 B

请求 C

气泡

"Static Batching 静态批处理 — 大量气泡"

Continuous Batching — 零气泡：

#mermaid-svg-7bRGRgzgdLCd16h3{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-7bRGRgzgdLCd16h3 .error-icon{fill:#552222;}#mermaid-svg-7bRGRgzgdLCd16h3 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-7bRGRgzgdLCd16h3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-7bRGRgzgdLCd16h3 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-7bRGRgzgdLCd16h3 .marker.cross{stroke:#333333;}#mermaid-svg-7bRGRgzgdLCd16h3 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-7bRGRgzgdLCd16h3 p{margin:0;}#mermaid-svg-7bRGRgzgdLCd16h3 .mermaid-main-font{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-7bRGRgzgdLCd16h3 .exclude-range{fill:#eeeeee;}#mermaid-svg-7bRGRgzgdLCd16h3 .section{stroke:none;opacity:0.2;}#mermaid-svg-7bRGRgzgdLCd16h3 .section0{fill:rgba(102, 102, 255, 0.49);}#mermaid-svg-7bRGRgzgdLCd16h3 .section2{fill:#fff400;}#mermaid-svg-7bRGRgzgdLCd16h3 .section1,#mermaid-svg-7bRGRgzgdLCd16h3 .section3{fill:white;opacity:0.2;}#mermaid-svg-7bRGRgzgdLCd16h3 .sectionTitle0{fill:#333;}#mermaid-svg-7bRGRgzgdLCd16h3 .sectionTitle1{fill:#333;}#mermaid-svg-7bRGRgzgdLCd16h3 .sectionTitle2{fill:#333;}#mermaid-svg-7bRGRgzgdLCd16h3 .sectionTitle3{fill:#333;}#mermaid-svg-7bRGRgzgdLCd16h3 .sectionTitle{text-anchor:start;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-7bRGRgzgdLCd16h3 .grid .tick{stroke:lightgrey;opacity:0.8;shape-rendering:crispEdges;}#mermaid-svg-7bRGRgzgdLCd16h3 .grid .tick text{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;fill:#333;}#mermaid-svg-7bRGRgzgdLCd16h3 .grid path{stroke-width:0;}#mermaid-svg-7bRGRgzgdLCd16h3 .today{fill:none;stroke:red;stroke-width:2px;}#mermaid-svg-7bRGRgzgdLCd16h3 .task{stroke-width:2;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskText{text-anchor:middle;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutsideRight{fill:black;text-anchor:start;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutsideLeft{fill:black;text-anchor:end;}#mermaid-svg-7bRGRgzgdLCd16h3 .task.clickable{cursor:pointer;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskText.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskText0,#mermaid-svg-7bRGRgzgdLCd16h3 .taskText1,#mermaid-svg-7bRGRgzgdLCd16h3 .taskText2,#mermaid-svg-7bRGRgzgdLCd16h3 .taskText3{fill:white;}#mermaid-svg-7bRGRgzgdLCd16h3 .task0,#mermaid-svg-7bRGRgzgdLCd16h3 .task1,#mermaid-svg-7bRGRgzgdLCd16h3 .task2,#mermaid-svg-7bRGRgzgdLCd16h3 .task3{fill:#8a90dd;stroke:#534fbc;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutside0,#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutside2{fill:black;}#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutside1,#mermaid-svg-7bRGRgzgdLCd16h3 .taskTextOutside3{fill:black;}#mermaid-svg-7bRGRgzgdLCd16h3 .active0,#mermaid-svg-7bRGRgzgdLCd16h3 .active1,#mermaid-svg-7bRGRgzgdLCd16h3 .active2,#mermaid-svg-7bRGRgzgdLCd16h3 .active3{fill:#bfc7ff;stroke:#534fbc;}#mermaid-svg-7bRGRgzgdLCd16h3 .activeText0,#mermaid-svg-7bRGRgzgdLCd16h3 .activeText1,#mermaid-svg-7bRGRgzgdLCd16h3 .activeText2,#mermaid-svg-7bRGRgzgdLCd16h3 .activeText3{fill:black!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .done0,#mermaid-svg-7bRGRgzgdLCd16h3 .done1,#mermaid-svg-7bRGRgzgdLCd16h3 .done2,#mermaid-svg-7bRGRgzgdLCd16h3 .done3{stroke:grey;fill:lightgrey;stroke-width:2;}#mermaid-svg-7bRGRgzgdLCd16h3 .doneText0,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText1,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText2,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText3{fill:black!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .doneText0.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText0.taskTextOutsideRight,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText1.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText1.taskTextOutsideRight,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText2.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText2.taskTextOutsideRight,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText3.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .crit0,#mermaid-svg-7bRGRgzgdLCd16h3 .crit1,#mermaid-svg-7bRGRgzgdLCd16h3 .crit2,#mermaid-svg-7bRGRgzgdLCd16h3 .crit3{stroke:#ff8888;fill:red;stroke-width:2;}#mermaid-svg-7bRGRgzgdLCd16h3 .activeCrit0,#mermaid-svg-7bRGRgzgdLCd16h3 .activeCrit1,#mermaid-svg-7bRGRgzgdLCd16h3 .activeCrit2,#mermaid-svg-7bRGRgzgdLCd16h3 .activeCrit3{stroke:#ff8888;fill:#bfc7ff;stroke-width:2;}#mermaid-svg-7bRGRgzgdLCd16h3 .doneCrit0,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCrit1,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCrit2,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCrit3{stroke:#ff8888;fill:lightgrey;stroke-width:2;cursor:pointer;shape-rendering:crispEdges;}#mermaid-svg-7bRGRgzgdLCd16h3 .milestone{transform:rotate(45deg) scale(0.8,0.8);}#mermaid-svg-7bRGRgzgdLCd16h3 .milestoneText{font-style:italic;}#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText0,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText1,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText2,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText3{fill:black!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText0.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText0.taskTextOutsideRight,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText1.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText1.taskTextOutsideRight,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText2.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText2.taskTextOutsideRight,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText3.taskTextOutsideLeft,#mermaid-svg-7bRGRgzgdLCd16h3 .doneCritText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .vert{stroke:navy;}#mermaid-svg-7bRGRgzgdLCd16h3 .vertText{font-size:15px;text-anchor:middle;fill:navy!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .activeCritText0,#mermaid-svg-7bRGRgzgdLCd16h3 .activeCritText1,#mermaid-svg-7bRGRgzgdLCd16h3 .activeCritText2,#mermaid-svg-7bRGRgzgdLCd16h3 .activeCritText3{fill:black!important;}#mermaid-svg-7bRGRgzgdLCd16h3 .titleText{text-anchor:middle;font-size:18px;fill:#333;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-7bRGRgzgdLCd16h3 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

0

0

1

1

2

2

3

3

4

4

5

Prefill A

Prefill B

Decode A1

Prefill C

Decode A2

Decode B1

Prefill D

Decode A3

Decode B2

Decode C1

请求 A

请求 B

请求 C

请求 D

"Continuous Batching — 零气泡"

实现 Continuous Batching 的关键：各请求的 KV Cache 在显存中独立管理——这正是 PagedAttention 的用武之地。

7.5 PagedAttention 与 vLLM

受操作系统虚拟内存分页机制启发：

#mermaid-svg-mVVEmLfugcr9tJ0l{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-mVVEmLfugcr9tJ0l .error-icon{fill:#552222;}#mermaid-svg-mVVEmLfugcr9tJ0l .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-mVVEmLfugcr9tJ0l .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-mVVEmLfugcr9tJ0l .marker{fill:#333333;stroke:#333333;}#mermaid-svg-mVVEmLfugcr9tJ0l .marker.cross{stroke:#333333;}#mermaid-svg-mVVEmLfugcr9tJ0l svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-mVVEmLfugcr9tJ0l p{margin:0;}#mermaid-svg-mVVEmLfugcr9tJ0l .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l .cluster-label text{fill:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l .cluster-label span{color:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l .cluster-label span p{background-color:transparent;}#mermaid-svg-mVVEmLfugcr9tJ0l .label text,#mermaid-svg-mVVEmLfugcr9tJ0l span{fill:#333;color:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l .node rect,#mermaid-svg-mVVEmLfugcr9tJ0l .node circle,#mermaid-svg-mVVEmLfugcr9tJ0l .node ellipse,#mermaid-svg-mVVEmLfugcr9tJ0l .node polygon,#mermaid-svg-mVVEmLfugcr9tJ0l .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-mVVEmLfugcr9tJ0l .rough-node .label text,#mermaid-svg-mVVEmLfugcr9tJ0l .node .label text,#mermaid-svg-mVVEmLfugcr9tJ0l .image-shape .label,#mermaid-svg-mVVEmLfugcr9tJ0l .icon-shape .label{text-anchor:middle;}#mermaid-svg-mVVEmLfugcr9tJ0l .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-mVVEmLfugcr9tJ0l .rough-node .label,#mermaid-svg-mVVEmLfugcr9tJ0l .node .label,#mermaid-svg-mVVEmLfugcr9tJ0l .image-shape .label,#mermaid-svg-mVVEmLfugcr9tJ0l .icon-shape .label{text-align:center;}#mermaid-svg-mVVEmLfugcr9tJ0l .node.clickable{cursor:pointer;}#mermaid-svg-mVVEmLfugcr9tJ0l .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-mVVEmLfugcr9tJ0l .arrowheadPath{fill:#333333;}#mermaid-svg-mVVEmLfugcr9tJ0l .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-mVVEmLfugcr9tJ0l .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-mVVEmLfugcr9tJ0l .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-mVVEmLfugcr9tJ0l .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-mVVEmLfugcr9tJ0l .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-mVVEmLfugcr9tJ0l .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-mVVEmLfugcr9tJ0l .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-mVVEmLfugcr9tJ0l .cluster text{fill:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l .cluster span{color:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-mVVEmLfugcr9tJ0l .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-mVVEmLfugcr9tJ0l rect.text{fill:none;stroke-width:0;}#mermaid-svg-mVVEmLfugcr9tJ0l .icon-shape,#mermaid-svg-mVVEmLfugcr9tJ0l .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-mVVEmLfugcr9tJ0l .icon-shape p,#mermaid-svg-mVVEmLfugcr9tJ0l .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-mVVEmLfugcr9tJ0l .icon-shape .label rect,#mermaid-svg-mVVEmLfugcr9tJ0l .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-mVVEmLfugcr9tJ0l .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-mVVEmLfugcr9tJ0l .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-mVVEmLfugcr9tJ0l :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

PagedAttention：分页内存

请求 A: [块1] [块2] [块3] [块4]

请求 B: [块5] [块6]

请求 C: [块7] [块8] [块9]

传统：连续内存

请求 A KV Cache: [____________________]

请求 B KV Cache: [__________]

碎片: [________]

请求 C: 无法分配 (无足够连续空间)

核心概念：

逻辑 KV Cache → Block Table → 物理 KV Cache（离散的 block）

Block Table:
[逻辑块 0] → [物理块 17]
[逻辑块 1] → [物理块 42]
[逻辑块 2] → [物理块 8]
…

每个 block 通常包含 16 或 32 个 token 的 KV

7.6 Prefix Caching (RadixAttention)

多个请求共享相同 prompt 前缀时，复用 KV Cache：

#mermaid-svg-hMvkH45PEXcfDCI0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hMvkH45PEXcfDCI0 .error-icon{fill:#552222;}#mermaid-svg-hMvkH45PEXcfDCI0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hMvkH45PEXcfDCI0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hMvkH45PEXcfDCI0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hMvkH45PEXcfDCI0 .marker.cross{stroke:#333333;}#mermaid-svg-hMvkH45PEXcfDCI0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hMvkH45PEXcfDCI0 p{margin:0;}#mermaid-svg-hMvkH45PEXcfDCI0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 .cluster-label text{fill:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 .cluster-label span{color:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 .cluster-label span p{background-color:transparent;}#mermaid-svg-hMvkH45PEXcfDCI0 .label text,#mermaid-svg-hMvkH45PEXcfDCI0 span{fill:#333;color:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 .node rect,#mermaid-svg-hMvkH45PEXcfDCI0 .node circle,#mermaid-svg-hMvkH45PEXcfDCI0 .node ellipse,#mermaid-svg-hMvkH45PEXcfDCI0 .node polygon,#mermaid-svg-hMvkH45PEXcfDCI0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hMvkH45PEXcfDCI0 .rough-node .label text,#mermaid-svg-hMvkH45PEXcfDCI0 .node .label text,#mermaid-svg-hMvkH45PEXcfDCI0 .image-shape .label,#mermaid-svg-hMvkH45PEXcfDCI0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-hMvkH45PEXcfDCI0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hMvkH45PEXcfDCI0 .rough-node .label,#mermaid-svg-hMvkH45PEXcfDCI0 .node .label,#mermaid-svg-hMvkH45PEXcfDCI0 .image-shape .label,#mermaid-svg-hMvkH45PEXcfDCI0 .icon-shape .label{text-align:center;}#mermaid-svg-hMvkH45PEXcfDCI0 .node.clickable{cursor:pointer;}#mermaid-svg-hMvkH45PEXcfDCI0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hMvkH45PEXcfDCI0 .arrowheadPath{fill:#333333;}#mermaid-svg-hMvkH45PEXcfDCI0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hMvkH45PEXcfDCI0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hMvkH45PEXcfDCI0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hMvkH45PEXcfDCI0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hMvkH45PEXcfDCI0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hMvkH45PEXcfDCI0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hMvkH45PEXcfDCI0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hMvkH45PEXcfDCI0 .cluster text{fill:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 .cluster span{color:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hMvkH45PEXcfDCI0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hMvkH45PEXcfDCI0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-hMvkH45PEXcfDCI0 .icon-shape,#mermaid-svg-hMvkH45PEXcfDCI0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hMvkH45PEXcfDCI0 .icon-shape p,#mermaid-svg-hMvkH45PEXcfDCI0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hMvkH45PEXcfDCI0 .icon-shape .label rect,#mermaid-svg-hMvkH45PEXcfDCI0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hMvkH45PEXcfDCI0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hMvkH45PEXcfDCI0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hMvkH45PEXcfDCI0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Root

系统提示词: 你是一个AI助手…

请解释量子计算

请写一首诗

用户1

用户2

用户3

上图中，“系统提示词” 部分的 KV Cache 可以在所有请求间共享。如果系统提示词长度为 1000 token，每个请求可节省 ~1000 token 的 Prefill 时间。

7.7 CUDA Graph 优化

Decode 每步的计算图完全相同（只是输入数据不同）：

#mermaid-svg-xsoGdAHYA96B974E{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-xsoGdAHYA96B974E .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-xsoGdAHYA96B974E .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-xsoGdAHYA96B974E .error-icon{fill:#552222;}#mermaid-svg-xsoGdAHYA96B974E .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-xsoGdAHYA96B974E .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-xsoGdAHYA96B974E .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-xsoGdAHYA96B974E .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-xsoGdAHYA96B974E .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-xsoGdAHYA96B974E .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-xsoGdAHYA96B974E .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-xsoGdAHYA96B974E .marker{fill:#333333;stroke:#333333;}#mermaid-svg-xsoGdAHYA96B974E .marker.cross{stroke:#333333;}#mermaid-svg-xsoGdAHYA96B974E svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-xsoGdAHYA96B974E p{margin:0;}#mermaid-svg-xsoGdAHYA96B974E .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-xsoGdAHYA96B974E .cluster-label text{fill:#333;}#mermaid-svg-xsoGdAHYA96B974E .cluster-label span{color:#333;}#mermaid-svg-xsoGdAHYA96B974E .cluster-label span p{background-color:transparent;}#mermaid-svg-xsoGdAHYA96B974E .label text,#mermaid-svg-xsoGdAHYA96B974E span{fill:#333;color:#333;}#mermaid-svg-xsoGdAHYA96B974E .node rect,#mermaid-svg-xsoGdAHYA96B974E .node circle,#mermaid-svg-xsoGdAHYA96B974E .node ellipse,#mermaid-svg-xsoGdAHYA96B974E .node polygon,#mermaid-svg-xsoGdAHYA96B974E .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-xsoGdAHYA96B974E .rough-node .label text,#mermaid-svg-xsoGdAHYA96B974E .node .label text,#mermaid-svg-xsoGdAHYA96B974E .image-shape .label,#mermaid-svg-xsoGdAHYA96B974E .icon-shape .label{text-anchor:middle;}#mermaid-svg-xsoGdAHYA96B974E .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-xsoGdAHYA96B974E .rough-node .label,#mermaid-svg-xsoGdAHYA96B974E .node .label,#mermaid-svg-xsoGdAHYA96B974E .image-shape .label,#mermaid-svg-xsoGdAHYA96B974E .icon-shape .label{text-align:center;}#mermaid-svg-xsoGdAHYA96B974E .node.clickable{cursor:pointer;}#mermaid-svg-xsoGdAHYA96B974E .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-xsoGdAHYA96B974E .arrowheadPath{fill:#333333;}#mermaid-svg-xsoGdAHYA96B974E .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-xsoGdAHYA96B974E .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-xsoGdAHYA96B974E .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-xsoGdAHYA96B974E .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-xsoGdAHYA96B974E .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-xsoGdAHYA96B974E .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-xsoGdAHYA96B974E .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-xsoGdAHYA96B974E .cluster text{fill:#333;}#mermaid-svg-xsoGdAHYA96B974E .cluster span{color:#333;}#mermaid-svg-xsoGdAHYA96B974E div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-xsoGdAHYA96B974E .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-xsoGdAHYA96B974E rect.text{fill:none;stroke-width:0;}#mermaid-svg-xsoGdAHYA96B974E .icon-shape,#mermaid-svg-xsoGdAHYA96B974E .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-xsoGdAHYA96B974E .icon-shape p,#mermaid-svg-xsoGdAHYA96B974E .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-xsoGdAHYA96B974E .icon-shape .label rect,#mermaid-svg-xsoGdAHYA96B974E .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-xsoGdAHYA96B974E .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-xsoGdAHYA96B974E .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-xsoGdAHYA96B974E :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

CUDA Graph

一次图捕获

图重放 直接提交整个图

Kernel K

每步 ~5-10us, 10-30x 减少 Launch 开销

Eager 模式

Python 调度

Kernel Launch 1

Kernel Launch 2

…

Kernel Launch K 每步 ~100-300us Launch 开销

适用条件：

• 计算图固定（Decode 满足）
• batch size 固定
• 序列长度固定（需要 padding 或 attention mask）

7.8 Streaming LLM / 窗口 Attention

解决超长文本 Decode 时 KV Cache 爆炸的问题：

#mermaid-svg-hdEw3FKgo9dMaexr{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hdEw3FKgo9dMaexr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hdEw3FKgo9dMaexr .error-icon{fill:#552222;}#mermaid-svg-hdEw3FKgo9dMaexr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hdEw3FKgo9dMaexr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hdEw3FKgo9dMaexr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hdEw3FKgo9dMaexr .marker.cross{stroke:#333333;}#mermaid-svg-hdEw3FKgo9dMaexr svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hdEw3FKgo9dMaexr p{margin:0;}#mermaid-svg-hdEw3FKgo9dMaexr .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hdEw3FKgo9dMaexr .cluster-label text{fill:#333;}#mermaid-svg-hdEw3FKgo9dMaexr .cluster-label span{color:#333;}#mermaid-svg-hdEw3FKgo9dMaexr .cluster-label span p{background-color:transparent;}#mermaid-svg-hdEw3FKgo9dMaexr .label text,#mermaid-svg-hdEw3FKgo9dMaexr span{fill:#333;color:#333;}#mermaid-svg-hdEw3FKgo9dMaexr .node rect,#mermaid-svg-hdEw3FKgo9dMaexr .node circle,#mermaid-svg-hdEw3FKgo9dMaexr .node ellipse,#mermaid-svg-hdEw3FKgo9dMaexr .node polygon,#mermaid-svg-hdEw3FKgo9dMaexr .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hdEw3FKgo9dMaexr .rough-node .label text,#mermaid-svg-hdEw3FKgo9dMaexr .node .label text,#mermaid-svg-hdEw3FKgo9dMaexr .image-shape .label,#mermaid-svg-hdEw3FKgo9dMaexr .icon-shape .label{text-anchor:middle;}#mermaid-svg-hdEw3FKgo9dMaexr .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hdEw3FKgo9dMaexr .rough-node .label,#mermaid-svg-hdEw3FKgo9dMaexr .node .label,#mermaid-svg-hdEw3FKgo9dMaexr .image-shape .label,#mermaid-svg-hdEw3FKgo9dMaexr .icon-shape .label{text-align:center;}#mermaid-svg-hdEw3FKgo9dMaexr .node.clickable{cursor:pointer;}#mermaid-svg-hdEw3FKgo9dMaexr .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hdEw3FKgo9dMaexr .arrowheadPath{fill:#333333;}#mermaid-svg-hdEw3FKgo9dMaexr .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hdEw3FKgo9dMaexr .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hdEw3FKgo9dMaexr .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hdEw3FKgo9dMaexr .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hdEw3FKgo9dMaexr .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hdEw3FKgo9dMaexr .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hdEw3FKgo9dMaexr .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hdEw3FKgo9dMaexr .cluster text{fill:#333;}#mermaid-svg-hdEw3FKgo9dMaexr .cluster span{color:#333;}#mermaid-svg-hdEw3FKgo9dMaexr div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hdEw3FKgo9dMaexr .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hdEw3FKgo9dMaexr rect.text{fill:none;stroke-width:0;}#mermaid-svg-hdEw3FKgo9dMaexr .icon-shape,#mermaid-svg-hdEw3FKgo9dMaexr .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hdEw3FKgo9dMaexr .icon-shape p,#mermaid-svg-hdEw3FKgo9dMaexr .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hdEw3FKgo9dMaexr .icon-shape .label rect,#mermaid-svg-hdEw3FKgo9dMaexr .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hdEw3FKgo9dMaexr .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hdEw3FKgo9dMaexr .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hdEw3FKgo9dMaexr :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Streaming LLM: 滑动窗口 + 特殊 tokens

Token1 (Attention Sink) -> Token2 -> … -> 窗口[-W:] KV Cache = 2 + W * d

Standard: 全部保留

Token1 -> Token2 -> Token3 -> … -> Token100000 KV Cache = 100k * d

核心发现：前几个 token（Attention Sinks）虽然语义不重要，但数值上会吸收大量 attention 分数。必须保留它们，否则模型困惑度急剧下降。

7.9 Medusa / Eagle：并行 Decode 的另一种思路

不依赖草稿模型，直接在目标模型上添加多个预测头：

#mermaid-svg-UzysmraXZNmn97Q8{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-UzysmraXZNmn97Q8 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-UzysmraXZNmn97Q8 .error-icon{fill:#552222;}#mermaid-svg-UzysmraXZNmn97Q8 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-UzysmraXZNmn97Q8 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-UzysmraXZNmn97Q8 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-UzysmraXZNmn97Q8 .marker.cross{stroke:#333333;}#mermaid-svg-UzysmraXZNmn97Q8 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-UzysmraXZNmn97Q8 p{margin:0;}#mermaid-svg-UzysmraXZNmn97Q8 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-UzysmraXZNmn97Q8 .cluster-label text{fill:#333;}#mermaid-svg-UzysmraXZNmn97Q8 .cluster-label span{color:#333;}#mermaid-svg-UzysmraXZNmn97Q8 .cluster-label span p{background-color:transparent;}#mermaid-svg-UzysmraXZNmn97Q8 .label text,#mermaid-svg-UzysmraXZNmn97Q8 span{fill:#333;color:#333;}#mermaid-svg-UzysmraXZNmn97Q8 .node rect,#mermaid-svg-UzysmraXZNmn97Q8 .node circle,#mermaid-svg-UzysmraXZNmn97Q8 .node ellipse,#mermaid-svg-UzysmraXZNmn97Q8 .node polygon,#mermaid-svg-UzysmraXZNmn97Q8 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-UzysmraXZNmn97Q8 .rough-node .label text,#mermaid-svg-UzysmraXZNmn97Q8 .node .label text,#mermaid-svg-UzysmraXZNmn97Q8 .image-shape .label,#mermaid-svg-UzysmraXZNmn97Q8 .icon-shape .label{text-anchor:middle;}#mermaid-svg-UzysmraXZNmn97Q8 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-UzysmraXZNmn97Q8 .rough-node .label,#mermaid-svg-UzysmraXZNmn97Q8 .node .label,#mermaid-svg-UzysmraXZNmn97Q8 .image-shape .label,#mermaid-svg-UzysmraXZNmn97Q8 .icon-shape .label{text-align:center;}#mermaid-svg-UzysmraXZNmn97Q8 .node.clickable{cursor:pointer;}#mermaid-svg-UzysmraXZNmn97Q8 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-UzysmraXZNmn97Q8 .arrowheadPath{fill:#333333;}#mermaid-svg-UzysmraXZNmn97Q8 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-UzysmraXZNmn97Q8 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-UzysmraXZNmn97Q8 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-UzysmraXZNmn97Q8 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-UzysmraXZNmn97Q8 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-UzysmraXZNmn97Q8 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-UzysmraXZNmn97Q8 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-UzysmraXZNmn97Q8 .cluster text{fill:#333;}#mermaid-svg-UzysmraXZNmn97Q8 .cluster span{color:#333;}#mermaid-svg-UzysmraXZNmn97Q8 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-UzysmraXZNmn97Q8 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-UzysmraXZNmn97Q8 rect.text{fill:none;stroke-width:0;}#mermaid-svg-UzysmraXZNmn97Q8 .icon-shape,#mermaid-svg-UzysmraXZNmn97Q8 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-UzysmraXZNmn97Q8 .icon-shape p,#mermaid-svg-UzysmraXZNmn97Q8 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-UzysmraXZNmn97Q8 .icon-shape .label rect,#mermaid-svg-UzysmraXZNmn97Q8 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-UzysmraXZNmn97Q8 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-UzysmraXZNmn97Q8 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-UzysmraXZNmn97Q8 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Base Model 最后一层 hidden

Head 0 预测 token_{t}

Head 1 预测 token_{t+1}

Head 2 预测 token_{t+2}

Head 3 预测 token_{t+3}

组合所有可能的候选集

并行验证 Tree Attention

接受最长合理序列

🏗️ 8. 推理引擎架构对比

8.1 主流推理引擎的设计哲学

#mermaid-svg-c3142ptVQtkNZ31n{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-c3142ptVQtkNZ31n .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-c3142ptVQtkNZ31n .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-c3142ptVQtkNZ31n .error-icon{fill:#552222;}#mermaid-svg-c3142ptVQtkNZ31n .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-c3142ptVQtkNZ31n .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-c3142ptVQtkNZ31n .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-c3142ptVQtkNZ31n .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-c3142ptVQtkNZ31n .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-c3142ptVQtkNZ31n .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-c3142ptVQtkNZ31n .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-c3142ptVQtkNZ31n .marker{fill:#333333;stroke:#333333;}#mermaid-svg-c3142ptVQtkNZ31n .marker.cross{stroke:#333333;}#mermaid-svg-c3142ptVQtkNZ31n svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-c3142ptVQtkNZ31n p{margin:0;}#mermaid-svg-c3142ptVQtkNZ31n .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-c3142ptVQtkNZ31n .cluster-label text{fill:#333;}#mermaid-svg-c3142ptVQtkNZ31n .cluster-label span{color:#333;}#mermaid-svg-c3142ptVQtkNZ31n .cluster-label span p{background-color:transparent;}#mermaid-svg-c3142ptVQtkNZ31n .label text,#mermaid-svg-c3142ptVQtkNZ31n span{fill:#333;color:#333;}#mermaid-svg-c3142ptVQtkNZ31n .node rect,#mermaid-svg-c3142ptVQtkNZ31n .node circle,#mermaid-svg-c3142ptVQtkNZ31n .node ellipse,#mermaid-svg-c3142ptVQtkNZ31n .node polygon,#mermaid-svg-c3142ptVQtkNZ31n .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-c3142ptVQtkNZ31n .rough-node .label text,#mermaid-svg-c3142ptVQtkNZ31n .node .label text,#mermaid-svg-c3142ptVQtkNZ31n .image-shape .label,#mermaid-svg-c3142ptVQtkNZ31n .icon-shape .label{text-anchor:middle;}#mermaid-svg-c3142ptVQtkNZ31n .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-c3142ptVQtkNZ31n .rough-node .label,#mermaid-svg-c3142ptVQtkNZ31n .node .label,#mermaid-svg-c3142ptVQtkNZ31n .image-shape .label,#mermaid-svg-c3142ptVQtkNZ31n .icon-shape .label{text-align:center;}#mermaid-svg-c3142ptVQtkNZ31n .node.clickable{cursor:pointer;}#mermaid-svg-c3142ptVQtkNZ31n .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-c3142ptVQtkNZ31n .arrowheadPath{fill:#333333;}#mermaid-svg-c3142ptVQtkNZ31n .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-c3142ptVQtkNZ31n .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-c3142ptVQtkNZ31n .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-c3142ptVQtkNZ31n .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-c3142ptVQtkNZ31n .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-c3142ptVQtkNZ31n .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-c3142ptVQtkNZ31n .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-c3142ptVQtkNZ31n .cluster text{fill:#333;}#mermaid-svg-c3142ptVQtkNZ31n .cluster span{color:#333;}#mermaid-svg-c3142ptVQtkNZ31n div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-c3142ptVQtkNZ31n .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-c3142ptVQtkNZ31n rect.text{fill:none;stroke-width:0;}#mermaid-svg-c3142ptVQtkNZ31n .icon-shape,#mermaid-svg-c3142ptVQtkNZ31n .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-c3142ptVQtkNZ31n .icon-shape p,#mermaid-svg-c3142ptVQtkNZ31n .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-c3142ptVQtkNZ31n .icon-shape .label rect,#mermaid-svg-c3142ptVQtkNZ31n .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-c3142ptVQtkNZ31n .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-c3142ptVQtkNZ31n .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-c3142ptVQtkNZ31n :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

vLLM PagedAttention 内存优先 Python + CUDA 吞吐优先

TensorRT-LLM 图优化 + C++ 极致延迟优化 CUDA Graph NVIDIA 生态

LMDeploy TurboMind 4-bit KV Cache PyTorch 生态 国内优化

MLC-Engine TVM 编译优化 跨平台 Web/Mobile OpenCL/Vulkan

8.2 关键特性对比

8.3 Prefill-Decode 调度的实现差异

#mermaid-svg-jhVVmN0I8E37PiP5{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-jhVVmN0I8E37PiP5 .error-icon{fill:#552222;}#mermaid-svg-jhVVmN0I8E37PiP5 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-jhVVmN0I8E37PiP5 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-jhVVmN0I8E37PiP5 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-jhVVmN0I8E37PiP5 .marker.cross{stroke:#333333;}#mermaid-svg-jhVVmN0I8E37PiP5 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-jhVVmN0I8E37PiP5 p{margin:0;}#mermaid-svg-jhVVmN0I8E37PiP5 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 .cluster-label text{fill:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 .cluster-label span{color:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 .cluster-label span p{background-color:transparent;}#mermaid-svg-jhVVmN0I8E37PiP5 .label text,#mermaid-svg-jhVVmN0I8E37PiP5 span{fill:#333;color:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 .node rect,#mermaid-svg-jhVVmN0I8E37PiP5 .node circle,#mermaid-svg-jhVVmN0I8E37PiP5 .node ellipse,#mermaid-svg-jhVVmN0I8E37PiP5 .node polygon,#mermaid-svg-jhVVmN0I8E37PiP5 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-jhVVmN0I8E37PiP5 .rough-node .label text,#mermaid-svg-jhVVmN0I8E37PiP5 .node .label text,#mermaid-svg-jhVVmN0I8E37PiP5 .image-shape .label,#mermaid-svg-jhVVmN0I8E37PiP5 .icon-shape .label{text-anchor:middle;}#mermaid-svg-jhVVmN0I8E37PiP5 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-jhVVmN0I8E37PiP5 .rough-node .label,#mermaid-svg-jhVVmN0I8E37PiP5 .node .label,#mermaid-svg-jhVVmN0I8E37PiP5 .image-shape .label,#mermaid-svg-jhVVmN0I8E37PiP5 .icon-shape .label{text-align:center;}#mermaid-svg-jhVVmN0I8E37PiP5 .node.clickable{cursor:pointer;}#mermaid-svg-jhVVmN0I8E37PiP5 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-jhVVmN0I8E37PiP5 .arrowheadPath{fill:#333333;}#mermaid-svg-jhVVmN0I8E37PiP5 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-jhVVmN0I8E37PiP5 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-jhVVmN0I8E37PiP5 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-jhVVmN0I8E37PiP5 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-jhVVmN0I8E37PiP5 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-jhVVmN0I8E37PiP5 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-jhVVmN0I8E37PiP5 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-jhVVmN0I8E37PiP5 .cluster text{fill:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 .cluster span{color:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-jhVVmN0I8E37PiP5 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-jhVVmN0I8E37PiP5 rect.text{fill:none;stroke-width:0;}#mermaid-svg-jhVVmN0I8E37PiP5 .icon-shape,#mermaid-svg-jhVVmN0I8E37PiP5 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-jhVVmN0I8E37PiP5 .icon-shape p,#mermaid-svg-jhVVmN0I8E37PiP5 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-jhVVmN0I8E37PiP5 .icon-shape .label rect,#mermaid-svg-jhVVmN0I8E37PiP5 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-jhVVmN0I8E37PiP5 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-jhVVmN0I8E37PiP5 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-jhVVmN0I8E37PiP5 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

TensorRT-LLM 调度

请求队列

Phase: Prefill, K-merging

Phase: Decode, 动态 batch

当有空闲 KV 槽位 插入新 Prefill

vLLM 调度

等待队列

选新请求做 Prefill

与 Decode 请求合并 batch

执行一轮 forward

产出 token / 检测完成

🔧 9. 工程实践的血泪教训

9.1 Prefill 常见陷阱

#mermaid-svg-Cn9oebCUls7O15iA{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Cn9oebCUls7O15iA .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Cn9oebCUls7O15iA .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Cn9oebCUls7O15iA .error-icon{fill:#552222;}#mermaid-svg-Cn9oebCUls7O15iA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Cn9oebCUls7O15iA .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Cn9oebCUls7O15iA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Cn9oebCUls7O15iA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Cn9oebCUls7O15iA .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Cn9oebCUls7O15iA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Cn9oebCUls7O15iA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Cn9oebCUls7O15iA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Cn9oebCUls7O15iA .marker.cross{stroke:#333333;}#mermaid-svg-Cn9oebCUls7O15iA svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Cn9oebCUls7O15iA p{margin:0;}#mermaid-svg-Cn9oebCUls7O15iA .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Cn9oebCUls7O15iA .cluster-label text{fill:#333;}#mermaid-svg-Cn9oebCUls7O15iA .cluster-label span{color:#333;}#mermaid-svg-Cn9oebCUls7O15iA .cluster-label span p{background-color:transparent;}#mermaid-svg-Cn9oebCUls7O15iA .label text,#mermaid-svg-Cn9oebCUls7O15iA span{fill:#333;color:#333;}#mermaid-svg-Cn9oebCUls7O15iA .node rect,#mermaid-svg-Cn9oebCUls7O15iA .node circle,#mermaid-svg-Cn9oebCUls7O15iA .node ellipse,#mermaid-svg-Cn9oebCUls7O15iA .node polygon,#mermaid-svg-Cn9oebCUls7O15iA .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Cn9oebCUls7O15iA .rough-node .label text,#mermaid-svg-Cn9oebCUls7O15iA .node .label text,#mermaid-svg-Cn9oebCUls7O15iA .image-shape .label,#mermaid-svg-Cn9oebCUls7O15iA .icon-shape .label{text-anchor:middle;}#mermaid-svg-Cn9oebCUls7O15iA .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Cn9oebCUls7O15iA .rough-node .label,#mermaid-svg-Cn9oebCUls7O15iA .node .label,#mermaid-svg-Cn9oebCUls7O15iA .image-shape .label,#mermaid-svg-Cn9oebCUls7O15iA .icon-shape .label{text-align:center;}#mermaid-svg-Cn9oebCUls7O15iA .node.clickable{cursor:pointer;}#mermaid-svg-Cn9oebCUls7O15iA .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Cn9oebCUls7O15iA .arrowheadPath{fill:#333333;}#mermaid-svg-Cn9oebCUls7O15iA .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Cn9oebCUls7O15iA .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Cn9oebCUls7O15iA .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Cn9oebCUls7O15iA .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Cn9oebCUls7O15iA .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Cn9oebCUls7O15iA .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Cn9oebCUls7O15iA .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Cn9oebCUls7O15iA .cluster text{fill:#333;}#mermaid-svg-Cn9oebCUls7O15iA .cluster span{color:#333;}#mermaid-svg-Cn9oebCUls7O15iA div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Cn9oebCUls7O15iA .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Cn9oebCUls7O15iA rect.text{fill:none;stroke-width:0;}#mermaid-svg-Cn9oebCUls7O15iA .icon-shape,#mermaid-svg-Cn9oebCUls7O15iA .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Cn9oebCUls7O15iA .icon-shape p,#mermaid-svg-Cn9oebCUls7O15iA .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Cn9oebCUls7O15iA .icon-shape .label rect,#mermaid-svg-Cn9oebCUls7O15iA .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Cn9oebCUls7O15iA .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Cn9oebCUls7O15iA .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Cn9oebCUls7O15iA :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

忽视长 prompt 的中间激活内存

使用 Flash Attention + chunked prefill

忽略 Prompt 中 Padding 的效率损失

动态 batch + 不等长处理

大量短请求按顺序一一 Prefill

合并短请求为更大的 batch

9.2 Decode 常见陷阱

#mermaid-svg-ZTBYNtROOIPZHoZG{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ZTBYNtROOIPZHoZG .error-icon{fill:#552222;}#mermaid-svg-ZTBYNtROOIPZHoZG .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ZTBYNtROOIPZHoZG .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ZTBYNtROOIPZHoZG .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ZTBYNtROOIPZHoZG .marker.cross{stroke:#333333;}#mermaid-svg-ZTBYNtROOIPZHoZG svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ZTBYNtROOIPZHoZG p{margin:0;}#mermaid-svg-ZTBYNtROOIPZHoZG .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG .cluster-label text{fill:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG .cluster-label span{color:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG .cluster-label span p{background-color:transparent;}#mermaid-svg-ZTBYNtROOIPZHoZG .label text,#mermaid-svg-ZTBYNtROOIPZHoZG span{fill:#333;color:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG .node rect,#mermaid-svg-ZTBYNtROOIPZHoZG .node circle,#mermaid-svg-ZTBYNtROOIPZHoZG .node ellipse,#mermaid-svg-ZTBYNtROOIPZHoZG .node polygon,#mermaid-svg-ZTBYNtROOIPZHoZG .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ZTBYNtROOIPZHoZG .rough-node .label text,#mermaid-svg-ZTBYNtROOIPZHoZG .node .label text,#mermaid-svg-ZTBYNtROOIPZHoZG .image-shape .label,#mermaid-svg-ZTBYNtROOIPZHoZG .icon-shape .label{text-anchor:middle;}#mermaid-svg-ZTBYNtROOIPZHoZG .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ZTBYNtROOIPZHoZG .rough-node .label,#mermaid-svg-ZTBYNtROOIPZHoZG .node .label,#mermaid-svg-ZTBYNtROOIPZHoZG .image-shape .label,#mermaid-svg-ZTBYNtROOIPZHoZG .icon-shape .label{text-align:center;}#mermaid-svg-ZTBYNtROOIPZHoZG .node.clickable{cursor:pointer;}#mermaid-svg-ZTBYNtROOIPZHoZG .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ZTBYNtROOIPZHoZG .arrowheadPath{fill:#333333;}#mermaid-svg-ZTBYNtROOIPZHoZG .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ZTBYNtROOIPZHoZG .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ZTBYNtROOIPZHoZG .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZTBYNtROOIPZHoZG .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ZTBYNtROOIPZHoZG .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZTBYNtROOIPZHoZG .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ZTBYNtROOIPZHoZG .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ZTBYNtROOIPZHoZG .cluster text{fill:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG .cluster span{color:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ZTBYNtROOIPZHoZG .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ZTBYNtROOIPZHoZG rect.text{fill:none;stroke-width:0;}#mermaid-svg-ZTBYNtROOIPZHoZG .icon-shape,#mermaid-svg-ZTBYNtROOIPZHoZG .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZTBYNtROOIPZHoZG .icon-shape p,#mermaid-svg-ZTBYNtROOIPZHoZG .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ZTBYNtROOIPZHoZG .icon-shape .label rect,#mermaid-svg-ZTBYNtROOIPZHoZG .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZTBYNtROOIPZHoZG .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ZTBYNtROOIPZHoZG .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ZTBYNtROOIPZHoZG :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

使用 Eager 模式执行 Decode

启用 CUDA Graph

在 Decode 中使用 FP16 精度

使用 FP8 / INT8 量化 KV Cache

按请求逐个 Decode (串行)

尽可能增大 Decode batch size

忽视 Attention 实现的选择

使用 FlashInfer / FlashAttention 后端

9.3 性能调优清单

🔭 10. 未来之路

10.1 当前架构的极限

#mermaid-svg-g9EtjPHSMYRhFUJb{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-g9EtjPHSMYRhFUJb .error-icon{fill:#552222;}#mermaid-svg-g9EtjPHSMYRhFUJb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-g9EtjPHSMYRhFUJb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-g9EtjPHSMYRhFUJb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-g9EtjPHSMYRhFUJb .marker.cross{stroke:#333333;}#mermaid-svg-g9EtjPHSMYRhFUJb svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-g9EtjPHSMYRhFUJb p{margin:0;}#mermaid-svg-g9EtjPHSMYRhFUJb .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb .cluster-label text{fill:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb .cluster-label span{color:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb .cluster-label span p{background-color:transparent;}#mermaid-svg-g9EtjPHSMYRhFUJb .label text,#mermaid-svg-g9EtjPHSMYRhFUJb span{fill:#333;color:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb .node rect,#mermaid-svg-g9EtjPHSMYRhFUJb .node circle,#mermaid-svg-g9EtjPHSMYRhFUJb .node ellipse,#mermaid-svg-g9EtjPHSMYRhFUJb .node polygon,#mermaid-svg-g9EtjPHSMYRhFUJb .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-g9EtjPHSMYRhFUJb .rough-node .label text,#mermaid-svg-g9EtjPHSMYRhFUJb .node .label text,#mermaid-svg-g9EtjPHSMYRhFUJb .image-shape .label,#mermaid-svg-g9EtjPHSMYRhFUJb .icon-shape .label{text-anchor:middle;}#mermaid-svg-g9EtjPHSMYRhFUJb .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-g9EtjPHSMYRhFUJb .rough-node .label,#mermaid-svg-g9EtjPHSMYRhFUJb .node .label,#mermaid-svg-g9EtjPHSMYRhFUJb .image-shape .label,#mermaid-svg-g9EtjPHSMYRhFUJb .icon-shape .label{text-align:center;}#mermaid-svg-g9EtjPHSMYRhFUJb .node.clickable{cursor:pointer;}#mermaid-svg-g9EtjPHSMYRhFUJb .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-g9EtjPHSMYRhFUJb .arrowheadPath{fill:#333333;}#mermaid-svg-g9EtjPHSMYRhFUJb .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-g9EtjPHSMYRhFUJb .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-g9EtjPHSMYRhFUJb .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-g9EtjPHSMYRhFUJb .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-g9EtjPHSMYRhFUJb .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-g9EtjPHSMYRhFUJb .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-g9EtjPHSMYRhFUJb .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-g9EtjPHSMYRhFUJb .cluster text{fill:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb .cluster span{color:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-g9EtjPHSMYRhFUJb .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-g9EtjPHSMYRhFUJb rect.text{fill:none;stroke-width:0;}#mermaid-svg-g9EtjPHSMYRhFUJb .icon-shape,#mermaid-svg-g9EtjPHSMYRhFUJb .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-g9EtjPHSMYRhFUJb .icon-shape p,#mermaid-svg-g9EtjPHSMYRhFUJb .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-g9EtjPHSMYRhFUJb .icon-shape .label rect,#mermaid-svg-g9EtjPHSMYRhFUJb .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-g9EtjPHSMYRhFUJb .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-g9EtjPHSMYRhFUJb .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-g9EtjPHSMYRhFUJb :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Prefill: O(n²) Attention, 可 Flash Attention 优化, 但计算量不减

Decode: O(n) 带宽受限, 每步计算密度 <10 FLOPs/Byte, 严重受限于 HBM 带宽

序列长度 n → ∞, KV Cache 显存 → ∞, Attention 时间 → ∞

10.2 有希望的突破方向

📖 附录：关键公式速查

# Prefill 单层 FLOPs
T_prefill_layer = 6nd² (QKV) + 4n²d (Attn) + 2nd² (Proj) + 4nd·d_ff (FFN)

# Decode 单步 FLOPs (第 t 步)
T_decode_step = 3d² (QKV) + 4td (Attn) + 2d² (Proj) + 4d·d_ff (FFN)
= O(d² + d·d_ff + td)

# KV Cache 大小
KV_cache = 2 × L × n × h_kv × d_k × sizeof(dtype)

# 加速比上限 (Amdahl Law)
S_max = 1 / (1 – p)
其中 p 是可并行化的比例（Prefill = 整个序列，Decode = 单步不可并行）

# Speculative Decoding 期望产出
E[N] = (1 – p_accept^{γ+1}) / (1 – p_accept)

🎯 结语

#mermaid-svg-U0GblrXF3I6zJSPb{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-U0GblrXF3I6zJSPb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-U0GblrXF3I6zJSPb .error-icon{fill:#552222;}#mermaid-svg-U0GblrXF3I6zJSPb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-U0GblrXF3I6zJSPb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-U0GblrXF3I6zJSPb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-U0GblrXF3I6zJSPb .marker.cross{stroke:#333333;}#mermaid-svg-U0GblrXF3I6zJSPb svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-U0GblrXF3I6zJSPb p{margin:0;}#mermaid-svg-U0GblrXF3I6zJSPb .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-U0GblrXF3I6zJSPb .cluster-label text{fill:#333;}#mermaid-svg-U0GblrXF3I6zJSPb .cluster-label span{color:#333;}#mermaid-svg-U0GblrXF3I6zJSPb .cluster-label span p{background-color:transparent;}#mermaid-svg-U0GblrXF3I6zJSPb .label text,#mermaid-svg-U0GblrXF3I6zJSPb span{fill:#333;color:#333;}#mermaid-svg-U0GblrXF3I6zJSPb .node rect,#mermaid-svg-U0GblrXF3I6zJSPb .node circle,#mermaid-svg-U0GblrXF3I6zJSPb .node ellipse,#mermaid-svg-U0GblrXF3I6zJSPb .node polygon,#mermaid-svg-U0GblrXF3I6zJSPb .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-U0GblrXF3I6zJSPb .rough-node .label text,#mermaid-svg-U0GblrXF3I6zJSPb .node .label text,#mermaid-svg-U0GblrXF3I6zJSPb .image-shape .label,#mermaid-svg-U0GblrXF3I6zJSPb .icon-shape .label{text-anchor:middle;}#mermaid-svg-U0GblrXF3I6zJSPb .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-U0GblrXF3I6zJSPb .rough-node .label,#mermaid-svg-U0GblrXF3I6zJSPb .node .label,#mermaid-svg-U0GblrXF3I6zJSPb .image-shape .label,#mermaid-svg-U0GblrXF3I6zJSPb .icon-shape .label{text-align:center;}#mermaid-svg-U0GblrXF3I6zJSPb .node.clickable{cursor:pointer;}#mermaid-svg-U0GblrXF3I6zJSPb .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-U0GblrXF3I6zJSPb .arrowheadPath{fill:#333333;}#mermaid-svg-U0GblrXF3I6zJSPb .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-U0GblrXF3I6zJSPb .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-U0GblrXF3I6zJSPb .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-U0GblrXF3I6zJSPb .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-U0GblrXF3I6zJSPb .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-U0GblrXF3I6zJSPb .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-U0GblrXF3I6zJSPb .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-U0GblrXF3I6zJSPb .cluster text{fill:#333;}#mermaid-svg-U0GblrXF3I6zJSPb .cluster span{color:#333;}#mermaid-svg-U0GblrXF3I6zJSPb div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-U0GblrXF3I6zJSPb .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-U0GblrXF3I6zJSPb rect.text{fill:none;stroke-width:0;}#mermaid-svg-U0GblrXF3I6zJSPb .icon-shape,#mermaid-svg-U0GblrXF3I6zJSPb .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-U0GblrXF3I6zJSPb .icon-shape p,#mermaid-svg-U0GblrXF3I6zJSPb .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-U0GblrXF3I6zJSPb .icon-shape .label rect,#mermaid-svg-U0GblrXF3I6zJSPb .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-U0GblrXF3I6zJSPb .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-U0GblrXF3I6zJSPb .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-U0GblrXF3I6zJSPb :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

理解 Prefill 和 Decode 的本质区别 🏭 Prefill = 计算密集型 = GEMM 🚌 Decode = 带宽密集型 = GEMV 优化思路完全不同: · 让 Prefill 算得更快 ← Flash Attention · 让 Decode 读得更少 ← KV Cache 量化 + 推测解码

Prefill 和 Decode 不是同一个问题的两种解法，而是两个不同的问题。 成功的推理系统，必定分别针对这两个阶段的物理瓶颈做出不同的设计选择——用正确的优化击中最短的木板。

🎯 本文图表使用标准 Mermaid 语法，可在 GitHub、Typora、Obsidian、VS Code + 插件等主流渲染器中正确渲染。

📝 最后更新：2026.06