Transformer与大语言模型：第16章 MoE：混合专家架构

第16章 MoE：混合专家架构

本章目标

理解现代大模型为什么要用 MoE（Mixture of Experts）：它到底怎么计算、为什么有效、为什么难训练、为什么需要负载均衡，以及为什么 DeepSeek、Mixtral 这样的模型能把 MoE 做得如此激进。

上一章我们看到，Qwen、Llama、DeepSeek 相比原始 Transformer 做了一系列改进：RoPE、RMSNorm、GQA、KV Cache、FlashAttention。这些改进大多是在优化 Attention。而本章要讲的 MoE，优化的是另一半——FFN。

MoE 是现代大模型（尤其是 DeepSeek 系列）最核心的架构创新之一。它回答了一个非常关键的问题：

能不能让模型拥有非常多的参数，但每个 Token 只使用其中一小部分？

16.1 从 Dense FFN 到稀疏专家

先回忆普通 Transformer 的 FFN。一个 Transformer Block 大致是：

#mermaid-svg-9dLeWmSZYC8OfPnj{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-9dLeWmSZYC8OfPnj .error-icon{fill:#552222;}#mermaid-svg-9dLeWmSZYC8OfPnj .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-9dLeWmSZYC8OfPnj .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-9dLeWmSZYC8OfPnj .marker{fill:#333333;stroke:#333333;}#mermaid-svg-9dLeWmSZYC8OfPnj .marker.cross{stroke:#333333;}#mermaid-svg-9dLeWmSZYC8OfPnj svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-9dLeWmSZYC8OfPnj p{margin:0;}#mermaid-svg-9dLeWmSZYC8OfPnj .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj .cluster-label text{fill:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj .cluster-label span{color:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj .cluster-label span p{background-color:transparent;}#mermaid-svg-9dLeWmSZYC8OfPnj .label text,#mermaid-svg-9dLeWmSZYC8OfPnj span{fill:#333;color:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj .node rect,#mermaid-svg-9dLeWmSZYC8OfPnj .node circle,#mermaid-svg-9dLeWmSZYC8OfPnj .node ellipse,#mermaid-svg-9dLeWmSZYC8OfPnj .node polygon,#mermaid-svg-9dLeWmSZYC8OfPnj .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-9dLeWmSZYC8OfPnj .rough-node .label text,#mermaid-svg-9dLeWmSZYC8OfPnj .node .label text,#mermaid-svg-9dLeWmSZYC8OfPnj .image-shape .label,#mermaid-svg-9dLeWmSZYC8OfPnj .icon-shape .label{text-anchor:middle;}#mermaid-svg-9dLeWmSZYC8OfPnj .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-9dLeWmSZYC8OfPnj .rough-node .label,#mermaid-svg-9dLeWmSZYC8OfPnj .node .label,#mermaid-svg-9dLeWmSZYC8OfPnj .image-shape .label,#mermaid-svg-9dLeWmSZYC8OfPnj .icon-shape .label{text-align:center;}#mermaid-svg-9dLeWmSZYC8OfPnj .node.clickable{cursor:pointer;}#mermaid-svg-9dLeWmSZYC8OfPnj .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-9dLeWmSZYC8OfPnj .arrowheadPath{fill:#333333;}#mermaid-svg-9dLeWmSZYC8OfPnj .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-9dLeWmSZYC8OfPnj .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-9dLeWmSZYC8OfPnj .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9dLeWmSZYC8OfPnj .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-9dLeWmSZYC8OfPnj .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9dLeWmSZYC8OfPnj .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-9dLeWmSZYC8OfPnj .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-9dLeWmSZYC8OfPnj .cluster text{fill:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj .cluster span{color:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-9dLeWmSZYC8OfPnj .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-9dLeWmSZYC8OfPnj rect.text{fill:none;stroke-width:0;}#mermaid-svg-9dLeWmSZYC8OfPnj .icon-shape,#mermaid-svg-9dLeWmSZYC8OfPnj .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9dLeWmSZYC8OfPnj .icon-shape p,#mermaid-svg-9dLeWmSZYC8OfPnj .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-9dLeWmSZYC8OfPnj .icon-shape .label rect,#mermaid-svg-9dLeWmSZYC8OfPnj .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9dLeWmSZYC8OfPnj .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-9dLeWmSZYC8OfPnj .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-9dLeWmSZYC8OfPnj :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Input

Attention

FFN

Output

Dense 模型里，所有 Token 都经过同一个 FFN：

#mermaid-svg-YQjOxKci1nF9xkil{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-YQjOxKci1nF9xkil .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-YQjOxKci1nF9xkil .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-YQjOxKci1nF9xkil .error-icon{fill:#552222;}#mermaid-svg-YQjOxKci1nF9xkil .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-YQjOxKci1nF9xkil .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-YQjOxKci1nF9xkil .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-YQjOxKci1nF9xkil .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-YQjOxKci1nF9xkil .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-YQjOxKci1nF9xkil .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-YQjOxKci1nF9xkil .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-YQjOxKci1nF9xkil .marker{fill:#333333;stroke:#333333;}#mermaid-svg-YQjOxKci1nF9xkil .marker.cross{stroke:#333333;}#mermaid-svg-YQjOxKci1nF9xkil svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-YQjOxKci1nF9xkil p{margin:0;}#mermaid-svg-YQjOxKci1nF9xkil .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-YQjOxKci1nF9xkil .cluster-label text{fill:#333;}#mermaid-svg-YQjOxKci1nF9xkil .cluster-label span{color:#333;}#mermaid-svg-YQjOxKci1nF9xkil .cluster-label span p{background-color:transparent;}#mermaid-svg-YQjOxKci1nF9xkil .label text,#mermaid-svg-YQjOxKci1nF9xkil span{fill:#333;color:#333;}#mermaid-svg-YQjOxKci1nF9xkil .node rect,#mermaid-svg-YQjOxKci1nF9xkil .node circle,#mermaid-svg-YQjOxKci1nF9xkil .node ellipse,#mermaid-svg-YQjOxKci1nF9xkil .node polygon,#mermaid-svg-YQjOxKci1nF9xkil .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-YQjOxKci1nF9xkil .rough-node .label text,#mermaid-svg-YQjOxKci1nF9xkil .node .label text,#mermaid-svg-YQjOxKci1nF9xkil .image-shape .label,#mermaid-svg-YQjOxKci1nF9xkil .icon-shape .label{text-anchor:middle;}#mermaid-svg-YQjOxKci1nF9xkil .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-YQjOxKci1nF9xkil .rough-node .label,#mermaid-svg-YQjOxKci1nF9xkil .node .label,#mermaid-svg-YQjOxKci1nF9xkil .image-shape .label,#mermaid-svg-YQjOxKci1nF9xkil .icon-shape .label{text-align:center;}#mermaid-svg-YQjOxKci1nF9xkil .node.clickable{cursor:pointer;}#mermaid-svg-YQjOxKci1nF9xkil .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-YQjOxKci1nF9xkil .arrowheadPath{fill:#333333;}#mermaid-svg-YQjOxKci1nF9xkil .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-YQjOxKci1nF9xkil .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-YQjOxKci1nF9xkil .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-YQjOxKci1nF9xkil .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-YQjOxKci1nF9xkil .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-YQjOxKci1nF9xkil .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-YQjOxKci1nF9xkil .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-YQjOxKci1nF9xkil .cluster text{fill:#333;}#mermaid-svg-YQjOxKci1nF9xkil .cluster span{color:#333;}#mermaid-svg-YQjOxKci1nF9xkil div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-YQjOxKci1nF9xkil .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-YQjOxKci1nF9xkil rect.text{fill:none;stroke-width:0;}#mermaid-svg-YQjOxKci1nF9xkil .icon-shape,#mermaid-svg-YQjOxKci1nF9xkil .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-YQjOxKci1nF9xkil .icon-shape p,#mermaid-svg-YQjOxKci1nF9xkil .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-YQjOxKci1nF9xkil .icon-shape .label rect,#mermaid-svg-YQjOxKci1nF9xkil .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-YQjOxKci1nF9xkil .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-YQjOxKci1nF9xkil .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-YQjOxKci1nF9xkil :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Token

一个 FFN

Output

MoE 的核心变化是：把这一个固定的 FFN，换成多个 FFN + 一个动态路由：

#mermaid-svg-Kw7bnfvyUCrgMOiQ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .error-icon{fill:#552222;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .marker.cross{stroke:#333333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ p{margin:0;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .cluster-label text{fill:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .cluster-label span{color:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .cluster-label span p{background-color:transparent;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .label text,#mermaid-svg-Kw7bnfvyUCrgMOiQ span{fill:#333;color:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .node rect,#mermaid-svg-Kw7bnfvyUCrgMOiQ .node circle,#mermaid-svg-Kw7bnfvyUCrgMOiQ .node ellipse,#mermaid-svg-Kw7bnfvyUCrgMOiQ .node polygon,#mermaid-svg-Kw7bnfvyUCrgMOiQ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .rough-node .label text,#mermaid-svg-Kw7bnfvyUCrgMOiQ .node .label text,#mermaid-svg-Kw7bnfvyUCrgMOiQ .image-shape .label,#mermaid-svg-Kw7bnfvyUCrgMOiQ .icon-shape .label{text-anchor:middle;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .rough-node .label,#mermaid-svg-Kw7bnfvyUCrgMOiQ .node .label,#mermaid-svg-Kw7bnfvyUCrgMOiQ .image-shape .label,#mermaid-svg-Kw7bnfvyUCrgMOiQ .icon-shape .label{text-align:center;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .node.clickable{cursor:pointer;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .arrowheadPath{fill:#333333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Kw7bnfvyUCrgMOiQ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Kw7bnfvyUCrgMOiQ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Kw7bnfvyUCrgMOiQ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .cluster text{fill:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .cluster span{color:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Kw7bnfvyUCrgMOiQ rect.text{fill:none;stroke-width:0;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .icon-shape,#mermaid-svg-Kw7bnfvyUCrgMOiQ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .icon-shape p,#mermaid-svg-Kw7bnfvyUCrgMOiQ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .icon-shape .label rect,#mermaid-svg-Kw7bnfvyUCrgMOiQ .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Kw7bnfvyUCrgMOiQ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Kw7bnfvyUCrgMOiQ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Kw7bnfvyUCrgMOiQ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Token

Router

Expert 1

Expert 2

Expert 3

…

加权求和

Output

用一句话概括这个变化：

一个固定 FFN → 多个 FFN + 动态路由

这里要特别强调一点，否则很容易产生误解：

MoE 通常不是把整个 Transformer 复制成多个模型，而主要是把 FFN / MLP 部分专家化。 Attention 部分通常保持不变。

所以不要把\”160 个专家\”理解成\”160 个完整的 Transformer\”，而是\”某个 FFN 层里有 160 个并列的 FFN 子网络\”。

16.2 Dense FFN 为什么会成为瓶颈？

传统 Dense FFN 的计算是：

$$\text{FFN}(x)=W_2\sigma (W_{1}x)$$

所有 Token 都经过同一套参数 $W_1,W_2$。也就是说：

每一个 Token 都使用同一套参数。

假设模型有 100B 参数，那么每个 Token 都必须经过对应的大量计算。这导致一个非常关键的绑定关系：

$$\boxed{\text{模型参数量}⟺\text{每 Token 计算量}}$$

想让模型更\”聪明\”（更多参数），每个 Token 的计算成本就会同比上升。这正是 Dense 架构的根本瓶颈。

16.3 MoE 的核心思想

MoE 把 FFN 复制成多个 Expert（Expert 1、Expert 2、……、Expert N），然后规定：

每个 Token 只选择少量 Expert。

例如总共 64 个 Expert，每个 Token 只激活 2 个：

#mermaid-svg-VXrVWgFnLf39x9a9{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-VXrVWgFnLf39x9a9 .error-icon{fill:#552222;}#mermaid-svg-VXrVWgFnLf39x9a9 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-VXrVWgFnLf39x9a9 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-VXrVWgFnLf39x9a9 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-VXrVWgFnLf39x9a9 .marker.cross{stroke:#333333;}#mermaid-svg-VXrVWgFnLf39x9a9 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-VXrVWgFnLf39x9a9 p{margin:0;}#mermaid-svg-VXrVWgFnLf39x9a9 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 .cluster-label text{fill:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 .cluster-label span{color:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 .cluster-label span p{background-color:transparent;}#mermaid-svg-VXrVWgFnLf39x9a9 .label text,#mermaid-svg-VXrVWgFnLf39x9a9 span{fill:#333;color:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 .node rect,#mermaid-svg-VXrVWgFnLf39x9a9 .node circle,#mermaid-svg-VXrVWgFnLf39x9a9 .node ellipse,#mermaid-svg-VXrVWgFnLf39x9a9 .node polygon,#mermaid-svg-VXrVWgFnLf39x9a9 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-VXrVWgFnLf39x9a9 .rough-node .label text,#mermaid-svg-VXrVWgFnLf39x9a9 .node .label text,#mermaid-svg-VXrVWgFnLf39x9a9 .image-shape .label,#mermaid-svg-VXrVWgFnLf39x9a9 .icon-shape .label{text-anchor:middle;}#mermaid-svg-VXrVWgFnLf39x9a9 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-VXrVWgFnLf39x9a9 .rough-node .label,#mermaid-svg-VXrVWgFnLf39x9a9 .node .label,#mermaid-svg-VXrVWgFnLf39x9a9 .image-shape .label,#mermaid-svg-VXrVWgFnLf39x9a9 .icon-shape .label{text-align:center;}#mermaid-svg-VXrVWgFnLf39x9a9 .node.clickable{cursor:pointer;}#mermaid-svg-VXrVWgFnLf39x9a9 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-VXrVWgFnLf39x9a9 .arrowheadPath{fill:#333333;}#mermaid-svg-VXrVWgFnLf39x9a9 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-VXrVWgFnLf39x9a9 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-VXrVWgFnLf39x9a9 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-VXrVWgFnLf39x9a9 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-VXrVWgFnLf39x9a9 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-VXrVWgFnLf39x9a9 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-VXrVWgFnLf39x9a9 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-VXrVWgFnLf39x9a9 .cluster text{fill:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 .cluster span{color:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-VXrVWgFnLf39x9a9 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-VXrVWgFnLf39x9a9 rect.text{fill:none;stroke-width:0;}#mermaid-svg-VXrVWgFnLf39x9a9 .icon-shape,#mermaid-svg-VXrVWgFnLf39x9a9 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-VXrVWgFnLf39x9a9 .icon-shape p,#mermaid-svg-VXrVWgFnLf39x9a9 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-VXrVWgFnLf39x9a9 .icon-shape .label rect,#mermaid-svg-VXrVWgFnLf39x9a9 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-VXrVWgFnLf39x9a9 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-VXrVWgFnLf39x9a9 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-VXrVWgFnLf39x9a9 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

选中

选中

未选

一个 Token

Router

Expert 3 ✅

Expert 5 ✅

其余 62 个 Expert 不计算

于是就得到 MoE 最重要的性质：

$$\boxed{\text{总参数量可以很大}\text{但每 Token 的计算量不需要同比增加}}$$

这就是\”参数规模\”和\”计算规模\”的解耦——本章后面会反复回到这个核心。

16.4 Router 到底在做什么？

Router（路由器）可以理解成：

决定\”这个 Token 应该交给哪些专家处理\”。

它的输入是一个 Token 的隐藏状态 $x\in {\mathbb{R}}^d$，通过一个小矩阵 $W_r$ 打分后做 softmax：

$$g(x)=\text{softmax}(x{W}_r)=[p_1,p_2,\dots ,p_N]$$

得到每个 Expert 的分数。例如：

如果 Top-K = 2，就选出分数最高的 E3 和 E5，最终输出是这两个 Expert 结果的加权和：

$$y=p_3{E}_3(x)+p_5{E}_5(x)$$

整个 Router 的流程可以画成：

#mermaid-svg-pCLoN6s8KuNaeyCZ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-pCLoN6s8KuNaeyCZ .error-icon{fill:#552222;}#mermaid-svg-pCLoN6s8KuNaeyCZ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-pCLoN6s8KuNaeyCZ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .marker.cross{stroke:#333333;}#mermaid-svg-pCLoN6s8KuNaeyCZ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-pCLoN6s8KuNaeyCZ p{margin:0;}#mermaid-svg-pCLoN6s8KuNaeyCZ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .cluster-label text{fill:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .cluster-label span{color:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .cluster-label span p{background-color:transparent;}#mermaid-svg-pCLoN6s8KuNaeyCZ .label text,#mermaid-svg-pCLoN6s8KuNaeyCZ span{fill:#333;color:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .node rect,#mermaid-svg-pCLoN6s8KuNaeyCZ .node circle,#mermaid-svg-pCLoN6s8KuNaeyCZ .node ellipse,#mermaid-svg-pCLoN6s8KuNaeyCZ .node polygon,#mermaid-svg-pCLoN6s8KuNaeyCZ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .rough-node .label text,#mermaid-svg-pCLoN6s8KuNaeyCZ .node .label text,#mermaid-svg-pCLoN6s8KuNaeyCZ .image-shape .label,#mermaid-svg-pCLoN6s8KuNaeyCZ .icon-shape .label{text-anchor:middle;}#mermaid-svg-pCLoN6s8KuNaeyCZ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .rough-node .label,#mermaid-svg-pCLoN6s8KuNaeyCZ .node .label,#mermaid-svg-pCLoN6s8KuNaeyCZ .image-shape .label,#mermaid-svg-pCLoN6s8KuNaeyCZ .icon-shape .label{text-align:center;}#mermaid-svg-pCLoN6s8KuNaeyCZ .node.clickable{cursor:pointer;}#mermaid-svg-pCLoN6s8KuNaeyCZ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .arrowheadPath{fill:#333333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pCLoN6s8KuNaeyCZ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-pCLoN6s8KuNaeyCZ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pCLoN6s8KuNaeyCZ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-pCLoN6s8KuNaeyCZ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .cluster text{fill:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ .cluster span{color:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-pCLoN6s8KuNaeyCZ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-pCLoN6s8KuNaeyCZ rect.text{fill:none;stroke-width:0;}#mermaid-svg-pCLoN6s8KuNaeyCZ .icon-shape,#mermaid-svg-pCLoN6s8KuNaeyCZ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pCLoN6s8KuNaeyCZ .icon-shape p,#mermaid-svg-pCLoN6s8KuNaeyCZ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-pCLoN6s8KuNaeyCZ .icon-shape .label rect,#mermaid-svg-pCLoN6s8KuNaeyCZ .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pCLoN6s8KuNaeyCZ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-pCLoN6s8KuNaeyCZ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-pCLoN6s8KuNaeyCZ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Token x ∈ R^d

Router: softmax(x·Wr)

N 个 Expert 分数

Top-K 选择

Expert 3

Expert 5

加权求和y = p3·E3 + p5·E5

16.5 MoE 在 Transformer 中的位置：每层独立的专家团队

前面讲的都是\”一个 MoE 层\”内部发生了什么。但一个完整的 Transformer 有几十个 Block，MoE 到底装在哪里、各层之间是什么关系？这是初学者最容易混淆的地方。

先说结论：

在典型的 Transformer-based MoE 模型中，MoE 通常是替换某些 Transformer Block 里的 FFN / MLP 子层；如果每个 Block 都采用 MoE，那么每个 Block 都有自己独立的一组 Experts 和自己的 Router。

16.5.1 普通 Transformer vs MoE Transformer

普通 Transformer 里，每个 Block 是 Attention + 一个 FFN。假设有 32 个 Block，这 32 个 FFN 本来就是不同的参数。MoE Transformer 只是把其中的 FFN 换成\”Router + 多个 Expert\”：

#mermaid-svg-wDTxMxCCJUn1X2BF{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wDTxMxCCJUn1X2BF .error-icon{fill:#552222;}#mermaid-svg-wDTxMxCCJUn1X2BF .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wDTxMxCCJUn1X2BF .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wDTxMxCCJUn1X2BF .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wDTxMxCCJUn1X2BF .marker.cross{stroke:#333333;}#mermaid-svg-wDTxMxCCJUn1X2BF svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wDTxMxCCJUn1X2BF p{margin:0;}#mermaid-svg-wDTxMxCCJUn1X2BF .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF .cluster-label text{fill:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF .cluster-label span{color:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF .cluster-label span p{background-color:transparent;}#mermaid-svg-wDTxMxCCJUn1X2BF .label text,#mermaid-svg-wDTxMxCCJUn1X2BF span{fill:#333;color:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF .node rect,#mermaid-svg-wDTxMxCCJUn1X2BF .node circle,#mermaid-svg-wDTxMxCCJUn1X2BF .node ellipse,#mermaid-svg-wDTxMxCCJUn1X2BF .node polygon,#mermaid-svg-wDTxMxCCJUn1X2BF .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wDTxMxCCJUn1X2BF .rough-node .label text,#mermaid-svg-wDTxMxCCJUn1X2BF .node .label text,#mermaid-svg-wDTxMxCCJUn1X2BF .image-shape .label,#mermaid-svg-wDTxMxCCJUn1X2BF .icon-shape .label{text-anchor:middle;}#mermaid-svg-wDTxMxCCJUn1X2BF .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-wDTxMxCCJUn1X2BF .rough-node .label,#mermaid-svg-wDTxMxCCJUn1X2BF .node .label,#mermaid-svg-wDTxMxCCJUn1X2BF .image-shape .label,#mermaid-svg-wDTxMxCCJUn1X2BF .icon-shape .label{text-align:center;}#mermaid-svg-wDTxMxCCJUn1X2BF .node.clickable{cursor:pointer;}#mermaid-svg-wDTxMxCCJUn1X2BF .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-wDTxMxCCJUn1X2BF .arrowheadPath{fill:#333333;}#mermaid-svg-wDTxMxCCJUn1X2BF .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-wDTxMxCCJUn1X2BF .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-wDTxMxCCJUn1X2BF .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wDTxMxCCJUn1X2BF .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-wDTxMxCCJUn1X2BF .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wDTxMxCCJUn1X2BF .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-wDTxMxCCJUn1X2BF .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-wDTxMxCCJUn1X2BF .cluster text{fill:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF .cluster span{color:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-wDTxMxCCJUn1X2BF .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-wDTxMxCCJUn1X2BF rect.text{fill:none;stroke-width:0;}#mermaid-svg-wDTxMxCCJUn1X2BF .icon-shape,#mermaid-svg-wDTxMxCCJUn1X2BF .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wDTxMxCCJUn1X2BF .icon-shape p,#mermaid-svg-wDTxMxCCJUn1X2BF .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-wDTxMxCCJUn1X2BF .icon-shape .label rect,#mermaid-svg-wDTxMxCCJUn1X2BF .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wDTxMxCCJUn1X2BF .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-wDTxMxCCJUn1X2BF .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-wDTxMxCCJUn1X2BF :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

MoE Transformer Block

Self-Attention

Router

Expert 1

Expert 2

… Expert 64

普通 Transformer Block