生产问题排查与性能瓶颈定位：日志、监控、链路追踪、压测与Arthas

生产问题最怕的不是报错，而是没人知道从哪下手。

一个接口突然变慢，用户说提交订单失败，监控开始报警，日志里一堆异常，CPU 又有点高。这个时候如果只是凭感觉去猜，很容易越查越乱。

生产排障要有顺序：先确认影响范围，再拿日志、监控、链路追踪定位层级，然后用压测或复现缩小问题，必要时用 Arthas 进入 JVM 内部观察，最后修复、验证、复盘。

先把问题说清楚

遇到生产问题，第一步不是打开代码，而是把问题描述清楚。

至少要回答这几个问题：

可以用这个流程先收口：

#mermaid-svg-zluAKJsQyIIFHT3y{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-zluAKJsQyIIFHT3y .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-zluAKJsQyIIFHT3y .error-icon{fill:#552222;}#mermaid-svg-zluAKJsQyIIFHT3y .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-zluAKJsQyIIFHT3y .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-zluAKJsQyIIFHT3y .marker{fill:#333333;stroke:#333333;}#mermaid-svg-zluAKJsQyIIFHT3y .marker.cross{stroke:#333333;}#mermaid-svg-zluAKJsQyIIFHT3y svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-zluAKJsQyIIFHT3y p{margin:0;}#mermaid-svg-zluAKJsQyIIFHT3y .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-zluAKJsQyIIFHT3y .cluster-label text{fill:#333;}#mermaid-svg-zluAKJsQyIIFHT3y .cluster-label span{color:#333;}#mermaid-svg-zluAKJsQyIIFHT3y .cluster-label span p{background-color:transparent;}#mermaid-svg-zluAKJsQyIIFHT3y .label text,#mermaid-svg-zluAKJsQyIIFHT3y span{fill:#333;color:#333;}#mermaid-svg-zluAKJsQyIIFHT3y .node rect,#mermaid-svg-zluAKJsQyIIFHT3y .node circle,#mermaid-svg-zluAKJsQyIIFHT3y .node ellipse,#mermaid-svg-zluAKJsQyIIFHT3y .node polygon,#mermaid-svg-zluAKJsQyIIFHT3y .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-zluAKJsQyIIFHT3y .rough-node .label text,#mermaid-svg-zluAKJsQyIIFHT3y .node .label text,#mermaid-svg-zluAKJsQyIIFHT3y .image-shape .label,#mermaid-svg-zluAKJsQyIIFHT3y .icon-shape .label{text-anchor:middle;}#mermaid-svg-zluAKJsQyIIFHT3y .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-zluAKJsQyIIFHT3y .rough-node .label,#mermaid-svg-zluAKJsQyIIFHT3y .node .label,#mermaid-svg-zluAKJsQyIIFHT3y .image-shape .label,#mermaid-svg-zluAKJsQyIIFHT3y .icon-shape .label{text-align:center;}#mermaid-svg-zluAKJsQyIIFHT3y .node.clickable{cursor:pointer;}#mermaid-svg-zluAKJsQyIIFHT3y .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-zluAKJsQyIIFHT3y .arrowheadPath{fill:#333333;}#mermaid-svg-zluAKJsQyIIFHT3y .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-zluAKJsQyIIFHT3y .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-zluAKJsQyIIFHT3y .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-zluAKJsQyIIFHT3y .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-zluAKJsQyIIFHT3y .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-zluAKJsQyIIFHT3y .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-zluAKJsQyIIFHT3y .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-zluAKJsQyIIFHT3y .cluster text{fill:#333;}#mermaid-svg-zluAKJsQyIIFHT3y .cluster span{color:#333;}#mermaid-svg-zluAKJsQyIIFHT3y div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-zluAKJsQyIIFHT3y .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-zluAKJsQyIIFHT3y rect.text{fill:none;stroke-width:0;}#mermaid-svg-zluAKJsQyIIFHT3y .icon-shape,#mermaid-svg-zluAKJsQyIIFHT3y .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-zluAKJsQyIIFHT3y .icon-shape p,#mermaid-svg-zluAKJsQyIIFHT3y .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-zluAKJsQyIIFHT3y .icon-shape .label rect,#mermaid-svg-zluAKJsQyIIFHT3y .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-zluAKJsQyIIFHT3y .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-zluAKJsQyIIFHT3y .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-zluAKJsQyIIFHT3y :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

收到告警或用户反馈

确认异常时间

确认影响范围

确认异常接口或业务动作

查看近期变更

进入日志、监控、链路追踪分析

不要一上来就问“是不是数据库慢”。数据库当然可能慢，但也可能是线程池打满、缓存击穿、远程服务超时、锁竞争、GC 停顿、发布引入了死循环。

排障最重要的是用证据缩小范围。

一条通用排障主线

生产问题可以按“现象、范围、层级、原因、修复、验证”来推进。

#mermaid-svg-IKJWpuuPsegucgZl{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IKJWpuuPsegucgZl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IKJWpuuPsegucgZl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IKJWpuuPsegucgZl .error-icon{fill:#552222;}#mermaid-svg-IKJWpuuPsegucgZl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IKJWpuuPsegucgZl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IKJWpuuPsegucgZl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IKJWpuuPsegucgZl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IKJWpuuPsegucgZl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IKJWpuuPsegucgZl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IKJWpuuPsegucgZl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IKJWpuuPsegucgZl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IKJWpuuPsegucgZl .marker.cross{stroke:#333333;}#mermaid-svg-IKJWpuuPsegucgZl svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IKJWpuuPsegucgZl p{margin:0;}#mermaid-svg-IKJWpuuPsegucgZl .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-IKJWpuuPsegucgZl .cluster-label text{fill:#333;}#mermaid-svg-IKJWpuuPsegucgZl .cluster-label span{color:#333;}#mermaid-svg-IKJWpuuPsegucgZl .cluster-label span p{background-color:transparent;}#mermaid-svg-IKJWpuuPsegucgZl .label text,#mermaid-svg-IKJWpuuPsegucgZl span{fill:#333;color:#333;}#mermaid-svg-IKJWpuuPsegucgZl .node rect,#mermaid-svg-IKJWpuuPsegucgZl .node circle,#mermaid-svg-IKJWpuuPsegucgZl .node ellipse,#mermaid-svg-IKJWpuuPsegucgZl .node polygon,#mermaid-svg-IKJWpuuPsegucgZl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-IKJWpuuPsegucgZl .rough-node .label text,#mermaid-svg-IKJWpuuPsegucgZl .node .label text,#mermaid-svg-IKJWpuuPsegucgZl .image-shape .label,#mermaid-svg-IKJWpuuPsegucgZl .icon-shape .label{text-anchor:middle;}#mermaid-svg-IKJWpuuPsegucgZl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-IKJWpuuPsegucgZl .rough-node .label,#mermaid-svg-IKJWpuuPsegucgZl .node .label,#mermaid-svg-IKJWpuuPsegucgZl .image-shape .label,#mermaid-svg-IKJWpuuPsegucgZl .icon-shape .label{text-align:center;}#mermaid-svg-IKJWpuuPsegucgZl .node.clickable{cursor:pointer;}#mermaid-svg-IKJWpuuPsegucgZl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-IKJWpuuPsegucgZl .arrowheadPath{fill:#333333;}#mermaid-svg-IKJWpuuPsegucgZl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-IKJWpuuPsegucgZl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-IKJWpuuPsegucgZl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IKJWpuuPsegucgZl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-IKJWpuuPsegucgZl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IKJWpuuPsegucgZl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-IKJWpuuPsegucgZl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-IKJWpuuPsegucgZl .cluster text{fill:#333;}#mermaid-svg-IKJWpuuPsegucgZl .cluster span{color:#333;}#mermaid-svg-IKJWpuuPsegucgZl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-IKJWpuuPsegucgZl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-IKJWpuuPsegucgZl rect.text{fill:none;stroke-width:0;}#mermaid-svg-IKJWpuuPsegucgZl .icon-shape,#mermaid-svg-IKJWpuuPsegucgZl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IKJWpuuPsegucgZl .icon-shape p,#mermaid-svg-IKJWpuuPsegucgZl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-IKJWpuuPsegucgZl .icon-shape .label rect,#mermaid-svg-IKJWpuuPsegucgZl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IKJWpuuPsegucgZl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-IKJWpuuPsegucgZl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-IKJWpuuPsegucgZl :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

现象

慢、错、超时、资源高

范围

单接口、单服务、单机器、全链路

层级

应用、JVM、数据库、缓存、消息队列、网络

原因

配置、代码、流量、数据、依赖、资源

修复

回滚、降级、扩容、限流、修代码

验证

指标恢复、错误下降、用户路径正常

这个顺序很笨，但很稳。它避免了一个常见问题：还没确认影响范围，就开始改代码；还没确认瓶颈层级，就开始加机器。

日志看什么

日志适合回答“发生了什么”。

排查生产问题时，先用时间范围和 traceId 收敛日志。

#mermaid-svg-8zSjxHXBtgZJnvXY{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-8zSjxHXBtgZJnvXY .error-icon{fill:#552222;}#mermaid-svg-8zSjxHXBtgZJnvXY .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8zSjxHXBtgZJnvXY .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8zSjxHXBtgZJnvXY .marker{fill:#333333;stroke:#333333;}#mermaid-svg-8zSjxHXBtgZJnvXY .marker.cross{stroke:#333333;}#mermaid-svg-8zSjxHXBtgZJnvXY svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-8zSjxHXBtgZJnvXY p{margin:0;}#mermaid-svg-8zSjxHXBtgZJnvXY .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY .cluster-label text{fill:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY .cluster-label span{color:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY .cluster-label span p{background-color:transparent;}#mermaid-svg-8zSjxHXBtgZJnvXY .label text,#mermaid-svg-8zSjxHXBtgZJnvXY span{fill:#333;color:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY .node rect,#mermaid-svg-8zSjxHXBtgZJnvXY .node circle,#mermaid-svg-8zSjxHXBtgZJnvXY .node ellipse,#mermaid-svg-8zSjxHXBtgZJnvXY .node polygon,#mermaid-svg-8zSjxHXBtgZJnvXY .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-8zSjxHXBtgZJnvXY .rough-node .label text,#mermaid-svg-8zSjxHXBtgZJnvXY .node .label text,#mermaid-svg-8zSjxHXBtgZJnvXY .image-shape .label,#mermaid-svg-8zSjxHXBtgZJnvXY .icon-shape .label{text-anchor:middle;}#mermaid-svg-8zSjxHXBtgZJnvXY .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-8zSjxHXBtgZJnvXY .rough-node .label,#mermaid-svg-8zSjxHXBtgZJnvXY .node .label,#mermaid-svg-8zSjxHXBtgZJnvXY .image-shape .label,#mermaid-svg-8zSjxHXBtgZJnvXY .icon-shape .label{text-align:center;}#mermaid-svg-8zSjxHXBtgZJnvXY .node.clickable{cursor:pointer;}#mermaid-svg-8zSjxHXBtgZJnvXY .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-8zSjxHXBtgZJnvXY .arrowheadPath{fill:#333333;}#mermaid-svg-8zSjxHXBtgZJnvXY .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-8zSjxHXBtgZJnvXY .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-8zSjxHXBtgZJnvXY .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8zSjxHXBtgZJnvXY .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-8zSjxHXBtgZJnvXY .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8zSjxHXBtgZJnvXY .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-8zSjxHXBtgZJnvXY .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-8zSjxHXBtgZJnvXY .cluster text{fill:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY .cluster span{color:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8zSjxHXBtgZJnvXY .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-8zSjxHXBtgZJnvXY rect.text{fill:none;stroke-width:0;}#mermaid-svg-8zSjxHXBtgZJnvXY .icon-shape,#mermaid-svg-8zSjxHXBtgZJnvXY .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8zSjxHXBtgZJnvXY .icon-shape p,#mermaid-svg-8zSjxHXBtgZJnvXY .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-8zSjxHXBtgZJnvXY .icon-shape .label rect,#mermaid-svg-8zSjxHXBtgZJnvXY .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8zSjxHXBtgZJnvXY .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-8zSjxHXBtgZJnvXY .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-8zSjxHXBtgZJnvXY :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

异常时间

日志检索条件

traceId 或业务主键

异常栈

关键业务日志

上下游调用日志

常见查询条件包括：

日志里重点看几类信息：

一个好的排障日志应该能回答：

如果日志只能看到“操作失败”，看不到业务主键和调用耗时，那不是排障能力差，而是日志本身没设计好。

监控看什么

监控适合回答“系统状态是否异常”。

排查时不要只看 CPU。CPU 高只是现象，不一定是根因。

#mermaid-svg-Z2P4tn7psXvVRyHi{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Z2P4tn7psXvVRyHi .error-icon{fill:#552222;}#mermaid-svg-Z2P4tn7psXvVRyHi .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Z2P4tn7psXvVRyHi .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Z2P4tn7psXvVRyHi .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Z2P4tn7psXvVRyHi .marker.cross{stroke:#333333;}#mermaid-svg-Z2P4tn7psXvVRyHi svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Z2P4tn7psXvVRyHi p{margin:0;}#mermaid-svg-Z2P4tn7psXvVRyHi .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi .cluster-label text{fill:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi .cluster-label span{color:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi .cluster-label span p{background-color:transparent;}#mermaid-svg-Z2P4tn7psXvVRyHi .label text,#mermaid-svg-Z2P4tn7psXvVRyHi span{fill:#333;color:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi .node rect,#mermaid-svg-Z2P4tn7psXvVRyHi .node circle,#mermaid-svg-Z2P4tn7psXvVRyHi .node ellipse,#mermaid-svg-Z2P4tn7psXvVRyHi .node polygon,#mermaid-svg-Z2P4tn7psXvVRyHi .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Z2P4tn7psXvVRyHi .rough-node .label text,#mermaid-svg-Z2P4tn7psXvVRyHi .node .label text,#mermaid-svg-Z2P4tn7psXvVRyHi .image-shape .label,#mermaid-svg-Z2P4tn7psXvVRyHi .icon-shape .label{text-anchor:middle;}#mermaid-svg-Z2P4tn7psXvVRyHi .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Z2P4tn7psXvVRyHi .rough-node .label,#mermaid-svg-Z2P4tn7psXvVRyHi .node .label,#mermaid-svg-Z2P4tn7psXvVRyHi .image-shape .label,#mermaid-svg-Z2P4tn7psXvVRyHi .icon-shape .label{text-align:center;}#mermaid-svg-Z2P4tn7psXvVRyHi .node.clickable{cursor:pointer;}#mermaid-svg-Z2P4tn7psXvVRyHi .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Z2P4tn7psXvVRyHi .arrowheadPath{fill:#333333;}#mermaid-svg-Z2P4tn7psXvVRyHi .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Z2P4tn7psXvVRyHi .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Z2P4tn7psXvVRyHi .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Z2P4tn7psXvVRyHi .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Z2P4tn7psXvVRyHi .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Z2P4tn7psXvVRyHi .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Z2P4tn7psXvVRyHi .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Z2P4tn7psXvVRyHi .cluster text{fill:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi .cluster span{color:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Z2P4tn7psXvVRyHi .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Z2P4tn7psXvVRyHi rect.text{fill:none;stroke-width:0;}#mermaid-svg-Z2P4tn7psXvVRyHi .icon-shape,#mermaid-svg-Z2P4tn7psXvVRyHi .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Z2P4tn7psXvVRyHi .icon-shape p,#mermaid-svg-Z2P4tn7psXvVRyHi .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Z2P4tn7psXvVRyHi .icon-shape .label rect,#mermaid-svg-Z2P4tn7psXvVRyHi .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Z2P4tn7psXvVRyHi .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Z2P4tn7psXvVRyHi .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Z2P4tn7psXvVRyHi :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

监控指标

系统资源

JVM

应用接口

中间件

CPU、内存、磁盘 IO、网络

堆内存、GC、线程数

QPS、响应时间、错误率

数据库、缓存、消息队列

常见指标可以这样看：

监控的核心不是看一个点，而是看趋势和关联。

例如接口变慢时，应该同时看：

#mermaid-svg-U9lAxwMsX4GHoIcK{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-U9lAxwMsX4GHoIcK .error-icon{fill:#552222;}#mermaid-svg-U9lAxwMsX4GHoIcK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-U9lAxwMsX4GHoIcK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-U9lAxwMsX4GHoIcK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-U9lAxwMsX4GHoIcK .marker.cross{stroke:#333333;}#mermaid-svg-U9lAxwMsX4GHoIcK svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-U9lAxwMsX4GHoIcK p{margin:0;}#mermaid-svg-U9lAxwMsX4GHoIcK .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK .cluster-label text{fill:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK .cluster-label span{color:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK .cluster-label span p{background-color:transparent;}#mermaid-svg-U9lAxwMsX4GHoIcK .label text,#mermaid-svg-U9lAxwMsX4GHoIcK span{fill:#333;color:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK .node rect,#mermaid-svg-U9lAxwMsX4GHoIcK .node circle,#mermaid-svg-U9lAxwMsX4GHoIcK .node ellipse,#mermaid-svg-U9lAxwMsX4GHoIcK .node polygon,#mermaid-svg-U9lAxwMsX4GHoIcK .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-U9lAxwMsX4GHoIcK .rough-node .label text,#mermaid-svg-U9lAxwMsX4GHoIcK .node .label text,#mermaid-svg-U9lAxwMsX4GHoIcK .image-shape .label,#mermaid-svg-U9lAxwMsX4GHoIcK .icon-shape .label{text-anchor:middle;}#mermaid-svg-U9lAxwMsX4GHoIcK .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-U9lAxwMsX4GHoIcK .rough-node .label,#mermaid-svg-U9lAxwMsX4GHoIcK .node .label,#mermaid-svg-U9lAxwMsX4GHoIcK .image-shape .label,#mermaid-svg-U9lAxwMsX4GHoIcK .icon-shape .label{text-align:center;}#mermaid-svg-U9lAxwMsX4GHoIcK .node.clickable{cursor:pointer;}#mermaid-svg-U9lAxwMsX4GHoIcK .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-U9lAxwMsX4GHoIcK .arrowheadPath{fill:#333333;}#mermaid-svg-U9lAxwMsX4GHoIcK .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-U9lAxwMsX4GHoIcK .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-U9lAxwMsX4GHoIcK .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-U9lAxwMsX4GHoIcK .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-U9lAxwMsX4GHoIcK .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-U9lAxwMsX4GHoIcK .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-U9lAxwMsX4GHoIcK .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-U9lAxwMsX4GHoIcK .cluster text{fill:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK .cluster span{color:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-U9lAxwMsX4GHoIcK .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-U9lAxwMsX4GHoIcK rect.text{fill:none;stroke-width:0;}#mermaid-svg-U9lAxwMsX4GHoIcK .icon-shape,#mermaid-svg-U9lAxwMsX4GHoIcK .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-U9lAxwMsX4GHoIcK .icon-shape p,#mermaid-svg-U9lAxwMsX4GHoIcK .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-U9lAxwMsX4GHoIcK .icon-shape .label rect,#mermaid-svg-U9lAxwMsX4GHoIcK .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-U9lAxwMsX4GHoIcK .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-U9lAxwMsX4GHoIcK .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-U9lAxwMsX4GHoIcK :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

接口响应时间升高

QPS 是否升高

错误率是否升高

CPU 是否升高

GC 是否异常

数据库连接池是否打满

下游接口是否变慢

如果 QPS 没变，CPU 没变，但数据库慢查询突然增加，很可能瓶颈在数据库。
如果 QPS 暴涨，线程池队列堆积，下游超时增多，问题可能是流量超过了服务承载能力。
如果响应时间呈周期性尖刺，同时 Full GC 频繁，就要重点看 JVM 内存。

链路追踪看什么

链路追踪适合回答“慢在哪里”。

一次请求经过多个服务时，单看日志很容易碎。链路追踪能把调用关系画出来，并显示每一段耗时。

"支付服务""优惠券服务""库存服务""订单服务""网关""支付服务""优惠券服务""库存服务""订单服务""网关"#mermaid-svg-RfEJdOaFfd6pVx7H{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-RfEJdOaFfd6pVx7H .error-icon{fill:#552222;}#mermaid-svg-RfEJdOaFfd6pVx7H .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-RfEJdOaFfd6pVx7H .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-RfEJdOaFfd6pVx7H .marker{fill:#333333;stroke:#333333;}#mermaid-svg-RfEJdOaFfd6pVx7H .marker.cross{stroke:#333333;}#mermaid-svg-RfEJdOaFfd6pVx7H svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-RfEJdOaFfd6pVx7H p{margin:0;}#mermaid-svg-RfEJdOaFfd6pVx7H .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-RfEJdOaFfd6pVx7H text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-RfEJdOaFfd6pVx7H .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-RfEJdOaFfd6pVx7H .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-RfEJdOaFfd6pVx7H #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-RfEJdOaFfd6pVx7H .sequenceNumber{fill:white;}#mermaid-svg-RfEJdOaFfd6pVx7H #sequencenumber{fill:#333;}#mermaid-svg-RfEJdOaFfd6pVx7H #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-RfEJdOaFfd6pVx7H .messageText{fill:#333;stroke:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-RfEJdOaFfd6pVx7H .labelText,#mermaid-svg-RfEJdOaFfd6pVx7H .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .loopText,#mermaid-svg-RfEJdOaFfd6pVx7H .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-RfEJdOaFfd6pVx7H .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-RfEJdOaFfd6pVx7H .noteText,#mermaid-svg-RfEJdOaFfd6pVx7H .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-RfEJdOaFfd6pVx7H .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-RfEJdOaFfd6pVx7H .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-RfEJdOaFfd6pVx7H .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-RfEJdOaFfd6pVx7H .actorPopupMenu{position:absolute;}#mermaid-svg-RfEJdOaFfd6pVx7H .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-RfEJdOaFfd6pVx7H .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-RfEJdOaFfd6pVx7H .actor-man circle,#mermaid-svg-RfEJdOaFfd6pVx7H line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-RfEJdOaFfd6pVx7H :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}"下单请求 30ms""扣减库存 80ms""锁定优惠券 600ms""创建支付单 70ms"

看到这条链路，就不用猜了。主要耗时在优惠券服务。

链路追踪通常关注：

常见工具有 SkyWalking、Zipkin、Jaeger、OpenTelemetry 体系。工具名字不重要，重要的是系统要能把日志、指标、调用链用同一个 traceId 关联起来。

#mermaid-svg-yThaQ6m9SlQ2Ysm1{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .error-icon{fill:#552222;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .marker.cross{stroke:#333333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 p{margin:0;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .cluster-label text{fill:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .cluster-label span{color:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .cluster-label span p{background-color:transparent;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .label text,#mermaid-svg-yThaQ6m9SlQ2Ysm1 span{fill:#333;color:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node rect,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node circle,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node ellipse,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node polygon,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .rough-node .label text,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node .label text,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .image-shape .label,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .icon-shape .label{text-anchor:middle;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .rough-node .label,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node .label,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .image-shape .label,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .icon-shape .label{text-align:center;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node.clickable{cursor:pointer;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .arrowheadPath{fill:#333333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .cluster text{fill:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .cluster span{color:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 rect.text{fill:none;stroke-width:0;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .icon-shape,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .icon-shape p,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .icon-shape .label rect,#mermaid-svg-yThaQ6m9SlQ2Ysm1 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-yThaQ6m9SlQ2Ysm1 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

traceId

日志

链路追踪

指标标签

看异常细节

看调用路径和耗时

看整体趋势

有了这种关联，排查体验会完全不同。

压测用来验证瓶颈

压测不是为了跑一个漂亮数字，而是为了知道系统在哪个压力点开始撑不住。

这里要先分清压测和生产排障的边界：压测通常在上线前或压测环境做，用来验证容量和找潜在瓶颈；生产排障是在问题已经发生时，用日志、监控、链路追踪、线程栈、Arthas 等手段收集证据。不要为了定位线上问题，直接在生产环境随意压测，那很容易把故障扩大。

常见指标包括：

这里有个容易混的点：响应时间和延迟不是完全一回事。

在 JMeter 里，elapsed time 更接近一次请求从发送前到完整响应接收后的耗时；latency 更接近从发送前到收到响应第一部分的时间。实际分析时，响应时间更适合看用户整体等待，延迟更适合看服务开始响应的速度。

压测流程可以这样做：

#mermaid-svg-JmfYvhtMnGYQAtmJ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-JmfYvhtMnGYQAtmJ .error-icon{fill:#552222;}#mermaid-svg-JmfYvhtMnGYQAtmJ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-JmfYvhtMnGYQAtmJ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .marker.cross{stroke:#333333;}#mermaid-svg-JmfYvhtMnGYQAtmJ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-JmfYvhtMnGYQAtmJ p{margin:0;}#mermaid-svg-JmfYvhtMnGYQAtmJ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .cluster-label text{fill:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .cluster-label span{color:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .cluster-label span p{background-color:transparent;}#mermaid-svg-JmfYvhtMnGYQAtmJ .label text,#mermaid-svg-JmfYvhtMnGYQAtmJ span{fill:#333;color:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .node rect,#mermaid-svg-JmfYvhtMnGYQAtmJ .node circle,#mermaid-svg-JmfYvhtMnGYQAtmJ .node ellipse,#mermaid-svg-JmfYvhtMnGYQAtmJ .node polygon,#mermaid-svg-JmfYvhtMnGYQAtmJ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .rough-node .label text,#mermaid-svg-JmfYvhtMnGYQAtmJ .node .label text,#mermaid-svg-JmfYvhtMnGYQAtmJ .image-shape .label,#mermaid-svg-JmfYvhtMnGYQAtmJ .icon-shape .label{text-anchor:middle;}#mermaid-svg-JmfYvhtMnGYQAtmJ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .rough-node .label,#mermaid-svg-JmfYvhtMnGYQAtmJ .node .label,#mermaid-svg-JmfYvhtMnGYQAtmJ .image-shape .label,#mermaid-svg-JmfYvhtMnGYQAtmJ .icon-shape .label{text-align:center;}#mermaid-svg-JmfYvhtMnGYQAtmJ .node.clickable{cursor:pointer;}#mermaid-svg-JmfYvhtMnGYQAtmJ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .arrowheadPath{fill:#333333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-JmfYvhtMnGYQAtmJ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-JmfYvhtMnGYQAtmJ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-JmfYvhtMnGYQAtmJ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-JmfYvhtMnGYQAtmJ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .cluster text{fill:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ .cluster span{color:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-JmfYvhtMnGYQAtmJ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-JmfYvhtMnGYQAtmJ rect.text{fill:none;stroke-width:0;}#mermaid-svg-JmfYvhtMnGYQAtmJ .icon-shape,#mermaid-svg-JmfYvhtMnGYQAtmJ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-JmfYvhtMnGYQAtmJ .icon-shape p,#mermaid-svg-JmfYvhtMnGYQAtmJ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-JmfYvhtMnGYQAtmJ .icon-shape .label rect,#mermaid-svg-JmfYvhtMnGYQAtmJ .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-JmfYvhtMnGYQAtmJ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-JmfYvhtMnGYQAtmJ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-JmfYvhtMnGYQAtmJ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

是

否

确定压测目标

准备压测环境和数据

设置并发、持续时间、阶梯增压

执行压测

观察响应时间、QPS、错误率

同时观察 CPU、内存、GC、数据库、缓存

是否出现瓶颈

定位瓶颈层级

继续加压或结束

优化后复测

压测时不要只看 JMeter 报告。客户端报告告诉你“请求表现如何”，服务端监控告诉你“为什么会这样”。

比如：

压测结论要落到可执行动作上：扩容、调线程池、改 SQL、加缓存、拆热点、限流、降级、异步化，或者修代码。

Arthas 适合查什么

Arthas 是 Java 线上诊断工具，适合在不重启应用的情况下观察 JVM 内部状态。

它常用于这些场景：

常用命令：

dashboard

查看 JVM、线程、内存、GC 等实时概览。

thread

查看 Java 线程信息。CPU 高时经常用它找热点线程。

thread -n 5

查看最忙的几个线程。

jad com.example.OrderService

反编译 JVM 里实际加载的类，确认线上运行的代码是不是你以为的版本。

trace com.example.OrderService createOrder

跟踪方法内部调用耗时，适合定位慢在哪个子调用。

watch com.example.OrderService createOrder "{params, returnObj, throwExp}" -x 3

观察方法入参、返回值和异常。

Arthas 很强，但不能乱用。trace、watch 这类命令会对目标类做字节码增强，生产环境使用时要精确指定类和方法，避免范围过大。用完要及时 reset 或 stop。

#mermaid-svg-krbV3VoDwZsd3I5X{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-krbV3VoDwZsd3I5X .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-krbV3VoDwZsd3I5X .error-icon{fill:#552222;}#mermaid-svg-krbV3VoDwZsd3I5X .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-krbV3VoDwZsd3I5X .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-krbV3VoDwZsd3I5X .marker{fill:#333333;stroke:#333333;}#mermaid-svg-krbV3VoDwZsd3I5X .marker.cross{stroke:#333333;}#mermaid-svg-krbV3VoDwZsd3I5X svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-krbV3VoDwZsd3I5X p{margin:0;}#mermaid-svg-krbV3VoDwZsd3I5X .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-krbV3VoDwZsd3I5X .cluster-label text{fill:#333;}#mermaid-svg-krbV3VoDwZsd3I5X .cluster-label span{color:#333;}#mermaid-svg-krbV3VoDwZsd3I5X .cluster-label span p{background-color:transparent;}#mermaid-svg-krbV3VoDwZsd3I5X .label text,#mermaid-svg-krbV3VoDwZsd3I5X span{fill:#333;color:#333;}#mermaid-svg-krbV3VoDwZsd3I5X .node rect,#mermaid-svg-krbV3VoDwZsd3I5X .node circle,#mermaid-svg-krbV3VoDwZsd3I5X .node ellipse,#mermaid-svg-krbV3VoDwZsd3I5X .node polygon,#mermaid-svg-krbV3VoDwZsd3I5X .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-krbV3VoDwZsd3I5X .rough-node .label text,#mermaid-svg-krbV3VoDwZsd3I5X .node .label text,#mermaid-svg-krbV3VoDwZsd3I5X .image-shape .label,#mermaid-svg-krbV3VoDwZsd3I5X .icon-shape .label{text-anchor:middle;}#mermaid-svg-krbV3VoDwZsd3I5X .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-krbV3VoDwZsd3I5X .rough-node .label,#mermaid-svg-krbV3VoDwZsd3I5X .node .label,#mermaid-svg-krbV3VoDwZsd3I5X .image-shape .label,#mermaid-svg-krbV3VoDwZsd3I5X .icon-shape .label{text-align:center;}#mermaid-svg-krbV3VoDwZsd3I5X .node.clickable{cursor:pointer;}#mermaid-svg-krbV3VoDwZsd3I5X .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-krbV3VoDwZsd3I5X .arrowheadPath{fill:#333333;}#mermaid-svg-krbV3VoDwZsd3I5X .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-krbV3VoDwZsd3I5X .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-krbV3VoDwZsd3I5X .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-krbV3VoDwZsd3I5X .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-krbV3VoDwZsd3I5X .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-krbV3VoDwZsd3I5X .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-krbV3VoDwZsd3I5X .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-krbV3VoDwZsd3I5X .cluster text{fill:#333;}#mermaid-svg-krbV3VoDwZsd3I5X .cluster span{color:#333;}#mermaid-svg-krbV3VoDwZsd3I5X div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-krbV3VoDwZsd3I5X .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-krbV3VoDwZsd3I5X rect.text{fill:none;stroke-width:0;}#mermaid-svg-krbV3VoDwZsd3I5X .icon-shape,#mermaid-svg-krbV3VoDwZsd3I5X .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-krbV3VoDwZsd3I5X .icon-shape p,#mermaid-svg-krbV3VoDwZsd3I5X .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-krbV3VoDwZsd3I5X .icon-shape .label rect,#mermaid-svg-krbV3VoDwZsd3I5X .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-krbV3VoDwZsd3I5X .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-krbV3VoDwZsd3I5X .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-krbV3VoDwZsd3I5X :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

CPU 高

方法慢

参数异常

代码版本疑问

需要 JVM 内部证据

问题类型

thread 定位热点线程

trace 查看调用耗时

watch 观察入参返回值

jad 反编译已加载类

形成证据

远程调试要谨慎

远程 debug 很好用，但它不适合随便用在生产环境。

原因很简单：断点可能让线程挂起，业务请求会被卡住；如果断在锁、事务、连接池相关位置，影响会被放大。更麻烦的是，生产数据是真实用户数据，调试过程还有权限和安全风险。

更合理的边界是：

生产环境更推荐用日志、监控、链路追踪、Arthas、线程栈、堆转储、慢查询等方式拿证据。

#mermaid-svg-lvuKxrxkAdqzS6vJ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-lvuKxrxkAdqzS6vJ .error-icon{fill:#552222;}#mermaid-svg-lvuKxrxkAdqzS6vJ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-lvuKxrxkAdqzS6vJ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .marker.cross{stroke:#333333;}#mermaid-svg-lvuKxrxkAdqzS6vJ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-lvuKxrxkAdqzS6vJ p{margin:0;}#mermaid-svg-lvuKxrxkAdqzS6vJ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .cluster-label text{fill:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .cluster-label span{color:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .cluster-label span p{background-color:transparent;}#mermaid-svg-lvuKxrxkAdqzS6vJ .label text,#mermaid-svg-lvuKxrxkAdqzS6vJ span{fill:#333;color:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .node rect,#mermaid-svg-lvuKxrxkAdqzS6vJ .node circle,#mermaid-svg-lvuKxrxkAdqzS6vJ .node ellipse,#mermaid-svg-lvuKxrxkAdqzS6vJ .node polygon,#mermaid-svg-lvuKxrxkAdqzS6vJ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .rough-node .label text,#mermaid-svg-lvuKxrxkAdqzS6vJ .node .label text,#mermaid-svg-lvuKxrxkAdqzS6vJ .image-shape .label,#mermaid-svg-lvuKxrxkAdqzS6vJ .icon-shape .label{text-anchor:middle;}#mermaid-svg-lvuKxrxkAdqzS6vJ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .rough-node .label,#mermaid-svg-lvuKxrxkAdqzS6vJ .node .label,#mermaid-svg-lvuKxrxkAdqzS6vJ .image-shape .label,#mermaid-svg-lvuKxrxkAdqzS6vJ .icon-shape .label{text-align:center;}#mermaid-svg-lvuKxrxkAdqzS6vJ .node.clickable{cursor:pointer;}#mermaid-svg-lvuKxrxkAdqzS6vJ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .arrowheadPath{fill:#333333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lvuKxrxkAdqzS6vJ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-lvuKxrxkAdqzS6vJ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lvuKxrxkAdqzS6vJ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-lvuKxrxkAdqzS6vJ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .cluster text{fill:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ .cluster span{color:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-lvuKxrxkAdqzS6vJ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-lvuKxrxkAdqzS6vJ rect.text{fill:none;stroke-width:0;}#mermaid-svg-lvuKxrxkAdqzS6vJ .icon-shape,#mermaid-svg-lvuKxrxkAdqzS6vJ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lvuKxrxkAdqzS6vJ .icon-shape p,#mermaid-svg-lvuKxrxkAdqzS6vJ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-lvuKxrxkAdqzS6vJ .icon-shape .label rect,#mermaid-svg-lvuKxrxkAdqzS6vJ .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lvuKxrxkAdqzS6vJ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-lvuKxrxkAdqzS6vJ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-lvuKxrxkAdqzS6vJ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

能

不能

生产问题

能否在非生产环境复现

测试或预发远程 debug

生产环境采集证据

日志、监控、链路追踪

Arthas、线程栈、堆转储

定位根因

如果极端情况下必须在生产环境打开诊断能力，也要满足几个条件：审批明确、范围极小、时间极短、有人值守、随时可回滚。能不用断点，就不用断点。

常见问题怎么定位

接口突然变慢

#mermaid-svg-paCSGNJf5waQqZfw{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-paCSGNJf5waQqZfw .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-paCSGNJf5waQqZfw .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-paCSGNJf5waQqZfw .error-icon{fill:#552222;}#mermaid-svg-paCSGNJf5waQqZfw .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-paCSGNJf5waQqZfw .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-paCSGNJf5waQqZfw .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-paCSGNJf5waQqZfw .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-paCSGNJf5waQqZfw .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-paCSGNJf5waQqZfw .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-paCSGNJf5waQqZfw .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-paCSGNJf5waQqZfw .marker{fill:#333333;stroke:#333333;}#mermaid-svg-paCSGNJf5waQqZfw .marker.cross{stroke:#333333;}#mermaid-svg-paCSGNJf5waQqZfw svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-paCSGNJf5waQqZfw p{margin:0;}#mermaid-svg-paCSGNJf5waQqZfw .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-paCSGNJf5waQqZfw .cluster-label text{fill:#333;}#mermaid-svg-paCSGNJf5waQqZfw .cluster-label span{color:#333;}#mermaid-svg-paCSGNJf5waQqZfw .cluster-label span p{background-color:transparent;}#mermaid-svg-paCSGNJf5waQqZfw .label text,#mermaid-svg-paCSGNJf5waQqZfw span{fill:#333;color:#333;}#mermaid-svg-paCSGNJf5waQqZfw .node rect,#mermaid-svg-paCSGNJf5waQqZfw .node circle,#mermaid-svg-paCSGNJf5waQqZfw .node ellipse,#mermaid-svg-paCSGNJf5waQqZfw .node polygon,#mermaid-svg-paCSGNJf5waQqZfw .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-paCSGNJf5waQqZfw .rough-node .label text,#mermaid-svg-paCSGNJf5waQqZfw .node .label text,#mermaid-svg-paCSGNJf5waQqZfw .image-shape .label,#mermaid-svg-paCSGNJf5waQqZfw .icon-shape .label{text-anchor:middle;}#mermaid-svg-paCSGNJf5waQqZfw .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-paCSGNJf5waQqZfw .rough-node .label,#mermaid-svg-paCSGNJf5waQqZfw .node .label,#mermaid-svg-paCSGNJf5waQqZfw .image-shape .label,#mermaid-svg-paCSGNJf5waQqZfw .icon-shape .label{text-align:center;}#mermaid-svg-paCSGNJf5waQqZfw .node.clickable{cursor:pointer;}#mermaid-svg-paCSGNJf5waQqZfw .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-paCSGNJf5waQqZfw .arrowheadPath{fill:#333333;}#mermaid-svg-paCSGNJf5waQqZfw .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-paCSGNJf5waQqZfw .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-paCSGNJf5waQqZfw .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-paCSGNJf5waQqZfw .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-paCSGNJf5waQqZfw .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-paCSGNJf5waQqZfw .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-paCSGNJf5waQqZfw .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-paCSGNJf5waQqZfw .cluster text{fill:#333;}#mermaid-svg-paCSGNJf5waQqZfw .cluster span{color:#333;}#mermaid-svg-paCSGNJf5waQqZfw div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-paCSGNJf5waQqZfw .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-paCSGNJf5waQqZfw rect.text{fill:none;stroke-width:0;}#mermaid-svg-paCSGNJf5waQqZfw .icon-shape,#mermaid-svg-paCSGNJf5waQqZfw .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-paCSGNJf5waQqZfw .icon-shape p,#mermaid-svg-paCSGNJf5waQqZfw .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-paCSGNJf5waQqZfw .icon-shape .label rect,#mermaid-svg-paCSGNJf5waQqZfw .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-paCSGNJf5waQqZfw .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-paCSGNJf5waQqZfw .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-paCSGNJf5waQqZfw :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

接口变慢

看响应时间趋势

看 QPS 是否变化

看链路追踪最慢节点

看慢节点日志

看该服务 CPU、GC、线程池、连接池

确认数据库、缓存或下游依赖

如果慢点在数据库，就查慢 SQL、索引、锁等待、连接池。
如果慢点在下游接口，就查下游错误率、超时配置、重试策略。
如果慢点在本服务，就查线程池、锁、CPU、GC、热点方法。

CPU 飙高

#mermaid-svg-IjUzVtkSMbLVrNWH{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IjUzVtkSMbLVrNWH .error-icon{fill:#552222;}#mermaid-svg-IjUzVtkSMbLVrNWH .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IjUzVtkSMbLVrNWH .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IjUzVtkSMbLVrNWH .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IjUzVtkSMbLVrNWH .marker.cross{stroke:#333333;}#mermaid-svg-IjUzVtkSMbLVrNWH svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IjUzVtkSMbLVrNWH p{margin:0;}#mermaid-svg-IjUzVtkSMbLVrNWH .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH .cluster-label text{fill:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH .cluster-label span{color:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH .cluster-label span p{background-color:transparent;}#mermaid-svg-IjUzVtkSMbLVrNWH .label text,#mermaid-svg-IjUzVtkSMbLVrNWH span{fill:#333;color:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH .node rect,#mermaid-svg-IjUzVtkSMbLVrNWH .node circle,#mermaid-svg-IjUzVtkSMbLVrNWH .node ellipse,#mermaid-svg-IjUzVtkSMbLVrNWH .node polygon,#mermaid-svg-IjUzVtkSMbLVrNWH .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-IjUzVtkSMbLVrNWH .rough-node .label text,#mermaid-svg-IjUzVtkSMbLVrNWH .node .label text,#mermaid-svg-IjUzVtkSMbLVrNWH .image-shape .label,#mermaid-svg-IjUzVtkSMbLVrNWH .icon-shape .label{text-anchor:middle;}#mermaid-svg-IjUzVtkSMbLVrNWH .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-IjUzVtkSMbLVrNWH .rough-node .label,#mermaid-svg-IjUzVtkSMbLVrNWH .node .label,#mermaid-svg-IjUzVtkSMbLVrNWH .image-shape .label,#mermaid-svg-IjUzVtkSMbLVrNWH .icon-shape .label{text-align:center;}#mermaid-svg-IjUzVtkSMbLVrNWH .node.clickable{cursor:pointer;}#mermaid-svg-IjUzVtkSMbLVrNWH .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-IjUzVtkSMbLVrNWH .arrowheadPath{fill:#333333;}#mermaid-svg-IjUzVtkSMbLVrNWH .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-IjUzVtkSMbLVrNWH .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-IjUzVtkSMbLVrNWH .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IjUzVtkSMbLVrNWH .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-IjUzVtkSMbLVrNWH .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IjUzVtkSMbLVrNWH .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-IjUzVtkSMbLVrNWH .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-IjUzVtkSMbLVrNWH .cluster text{fill:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH .cluster span{color:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-IjUzVtkSMbLVrNWH .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-IjUzVtkSMbLVrNWH rect.text{fill:none;stroke-width:0;}#mermaid-svg-IjUzVtkSMbLVrNWH .icon-shape,#mermaid-svg-IjUzVtkSMbLVrNWH .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IjUzVtkSMbLVrNWH .icon-shape p,#mermaid-svg-IjUzVtkSMbLVrNWH .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-IjUzVtkSMbLVrNWH .icon-shape .label rect,#mermaid-svg-IjUzVtkSMbLVrNWH .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IjUzVtkSMbLVrNWH .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-IjUzVtkSMbLVrNWH .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-IjUzVtkSMbLVrNWH :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

CPU 飙高

确认是哪台机器

查看进程 CPU

查看热点线程

转换线程 ID

查看线程栈

判断死循环、计算密集、频繁 GC 或锁竞争

可以用 Arthas 的 thread -n 5 看热点线程，也可以用系统命令配合线程栈。关键是找到具体线程正在执行的代码，而不是只说“CPU 高”。

内存持续上涨

#mermaid-svg-deaTDzYjl4wcS7pf{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-deaTDzYjl4wcS7pf .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-deaTDzYjl4wcS7pf .error-icon{fill:#552222;}#mermaid-svg-deaTDzYjl4wcS7pf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-deaTDzYjl4wcS7pf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-deaTDzYjl4wcS7pf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-deaTDzYjl4wcS7pf .marker.cross{stroke:#333333;}#mermaid-svg-deaTDzYjl4wcS7pf svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-deaTDzYjl4wcS7pf p{margin:0;}#mermaid-svg-deaTDzYjl4wcS7pf .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-deaTDzYjl4wcS7pf .cluster-label text{fill:#333;}#mermaid-svg-deaTDzYjl4wcS7pf .cluster-label span{color:#333;}#mermaid-svg-deaTDzYjl4wcS7pf .cluster-label span p{background-color:transparent;}#mermaid-svg-deaTDzYjl4wcS7pf .label text,#mermaid-svg-deaTDzYjl4wcS7pf span{fill:#333;color:#333;}#mermaid-svg-deaTDzYjl4wcS7pf .node rect,#mermaid-svg-deaTDzYjl4wcS7pf .node circle,#mermaid-svg-deaTDzYjl4wcS7pf .node ellipse,#mermaid-svg-deaTDzYjl4wcS7pf .node polygon,#mermaid-svg-deaTDzYjl4wcS7pf .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-deaTDzYjl4wcS7pf .rough-node .label text,#mermaid-svg-deaTDzYjl4wcS7pf .node .label text,#mermaid-svg-deaTDzYjl4wcS7pf .image-shape .label,#mermaid-svg-deaTDzYjl4wcS7pf .icon-shape .label{text-anchor:middle;}#mermaid-svg-deaTDzYjl4wcS7pf .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-deaTDzYjl4wcS7pf .rough-node .label,#mermaid-svg-deaTDzYjl4wcS7pf .node .label,#mermaid-svg-deaTDzYjl4wcS7pf .image-shape .label,#mermaid-svg-deaTDzYjl4wcS7pf .icon-shape .label{text-align:center;}#mermaid-svg-deaTDzYjl4wcS7pf .node.clickable{cursor:pointer;}#mermaid-svg-deaTDzYjl4wcS7pf .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-deaTDzYjl4wcS7pf .arrowheadPath{fill:#333333;}#mermaid-svg-deaTDzYjl4wcS7pf .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-deaTDzYjl4wcS7pf .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-deaTDzYjl4wcS7pf .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-deaTDzYjl4wcS7pf .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-deaTDzYjl4wcS7pf .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-deaTDzYjl4wcS7pf .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-deaTDzYjl4wcS7pf .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-deaTDzYjl4wcS7pf .cluster text{fill:#333;}#mermaid-svg-deaTDzYjl4wcS7pf .cluster span{color:#333;}#mermaid-svg-deaTDzYjl4wcS7pf div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-deaTDzYjl4wcS7pf .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-deaTDzYjl4wcS7pf rect.text{fill:none;stroke-width:0;}#mermaid-svg-deaTDzYjl4wcS7pf .icon-shape,#mermaid-svg-deaTDzYjl4wcS7pf .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-deaTDzYjl4wcS7pf .icon-shape p,#mermaid-svg-deaTDzYjl4wcS7pf .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-deaTDzYjl4wcS7pf .icon-shape .label rect,#mermaid-svg-deaTDzYjl4wcS7pf .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-deaTDzYjl4wcS7pf .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-deaTDzYjl4wcS7pf .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-deaTDzYjl4wcS7pf :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

内存持续上涨

看堆内存曲线

看 GC 是否能回收

确认对象是否持续增长

检查缓存、集合、队列、ThreadLocal

必要时分析堆转储

内存问题要区分：

数据库慢

数据库问题不要只看一条 SQL。

#mermaid-svg-hznx7Xn7KBWl9OLg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hznx7Xn7KBWl9OLg .error-icon{fill:#552222;}#mermaid-svg-hznx7Xn7KBWl9OLg .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hznx7Xn7KBWl9OLg .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hznx7Xn7KBWl9OLg .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hznx7Xn7KBWl9OLg .marker.cross{stroke:#333333;}#mermaid-svg-hznx7Xn7KBWl9OLg svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hznx7Xn7KBWl9OLg p{margin:0;}#mermaid-svg-hznx7Xn7KBWl9OLg .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg .cluster-label text{fill:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg .cluster-label span{color:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg .cluster-label span p{background-color:transparent;}#mermaid-svg-hznx7Xn7KBWl9OLg .label text,#mermaid-svg-hznx7Xn7KBWl9OLg span{fill:#333;color:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg .node rect,#mermaid-svg-hznx7Xn7KBWl9OLg .node circle,#mermaid-svg-hznx7Xn7KBWl9OLg .node ellipse,#mermaid-svg-hznx7Xn7KBWl9OLg .node polygon,#mermaid-svg-hznx7Xn7KBWl9OLg .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hznx7Xn7KBWl9OLg .rough-node .label text,#mermaid-svg-hznx7Xn7KBWl9OLg .node .label text,#mermaid-svg-hznx7Xn7KBWl9OLg .image-shape .label,#mermaid-svg-hznx7Xn7KBWl9OLg .icon-shape .label{text-anchor:middle;}#mermaid-svg-hznx7Xn7KBWl9OLg .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hznx7Xn7KBWl9OLg .rough-node .label,#mermaid-svg-hznx7Xn7KBWl9OLg .node .label,#mermaid-svg-hznx7Xn7KBWl9OLg .image-shape .label,#mermaid-svg-hznx7Xn7KBWl9OLg .icon-shape .label{text-align:center;}#mermaid-svg-hznx7Xn7KBWl9OLg .node.clickable{cursor:pointer;}#mermaid-svg-hznx7Xn7KBWl9OLg .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hznx7Xn7KBWl9OLg .arrowheadPath{fill:#333333;}#mermaid-svg-hznx7Xn7KBWl9OLg .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hznx7Xn7KBWl9OLg .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hznx7Xn7KBWl9OLg .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hznx7Xn7KBWl9OLg .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hznx7Xn7KBWl9OLg .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hznx7Xn7KBWl9OLg .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hznx7Xn7KBWl9OLg .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hznx7Xn7KBWl9OLg .cluster text{fill:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg .cluster span{color:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hznx7Xn7KBWl9OLg .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hznx7Xn7KBWl9OLg rect.text{fill:none;stroke-width:0;}#mermaid-svg-hznx7Xn7KBWl9OLg .icon-shape,#mermaid-svg-hznx7Xn7KBWl9OLg .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hznx7Xn7KBWl9OLg .icon-shape p,#mermaid-svg-hznx7Xn7KBWl9OLg .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hznx7Xn7KBWl9OLg .icon-shape .label rect,#mermaid-svg-hznx7Xn7KBWl9OLg .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hznx7Xn7KBWl9OLg .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hznx7Xn7KBWl9OLg .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hznx7Xn7KBWl9OLg :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

数据库慢

查看慢查询

检查执行计划

检查索引是否命中

检查锁等待和事务时间

检查连接池是否耗尽

结合业务流量判断

很多线上数据库问题不是 SQL 写错这么简单，而是流量上来后索引选择变差、事务持有时间过长、热点行竞争、连接池配置过小，或者应用重试把数据库打得更狠。

消息堆积

#mermaid-svg-8lcnZbKX5b76JvbF{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-8lcnZbKX5b76JvbF .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-8lcnZbKX5b76JvbF .error-icon{fill:#552222;}#mermaid-svg-8lcnZbKX5b76JvbF .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8lcnZbKX5b76JvbF .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8lcnZbKX5b76JvbF .marker{fill:#333333;stroke:#333333;}#mermaid-svg-8lcnZbKX5b76JvbF .marker.cross{stroke:#333333;}#mermaid-svg-8lcnZbKX5b76JvbF svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-8lcnZbKX5b76JvbF p{margin:0;}#mermaid-svg-8lcnZbKX5b76JvbF .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-8lcnZbKX5b76JvbF .cluster-label text{fill:#333;}#mermaid-svg-8lcnZbKX5b76JvbF .cluster-label span{color:#333;}#mermaid-svg-8lcnZbKX5b76JvbF .cluster-label span p{background-color:transparent;}#mermaid-svg-8lcnZbKX5b76JvbF .label text,#mermaid-svg-8lcnZbKX5b76JvbF span{fill:#333;color:#333;}#mermaid-svg-8lcnZbKX5b76JvbF .node rect,#mermaid-svg-8lcnZbKX5b76JvbF .node circle,#mermaid-svg-8lcnZbKX5b76JvbF .node ellipse,#mermaid-svg-8lcnZbKX5b76JvbF .node polygon,#mermaid-svg-8lcnZbKX5b76JvbF .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-8lcnZbKX5b76JvbF .rough-node .label text,#mermaid-svg-8lcnZbKX5b76JvbF .node .label text,#mermaid-svg-8lcnZbKX5b76JvbF .image-shape .label,#mermaid-svg-8lcnZbKX5b76JvbF .icon-shape .label{text-anchor:middle;}#mermaid-svg-8lcnZbKX5b76JvbF .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-8lcnZbKX5b76JvbF .rough-node .label,#mermaid-svg-8lcnZbKX5b76JvbF .node .label,#mermaid-svg-8lcnZbKX5b76JvbF .image-shape .label,#mermaid-svg-8lcnZbKX5b76JvbF .icon-shape .label{text-align:center;}#mermaid-svg-8lcnZbKX5b76JvbF .node.clickable{cursor:pointer;}#mermaid-svg-8lcnZbKX5b76JvbF .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-8lcnZbKX5b76JvbF .arrowheadPath{fill:#333333;}#mermaid-svg-8lcnZbKX5b76JvbF .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-8lcnZbKX5b76JvbF .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-8lcnZbKX5b76JvbF .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8lcnZbKX5b76JvbF .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-8lcnZbKX5b76JvbF .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8lcnZbKX5b76JvbF .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-8lcnZbKX5b76JvbF .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-8lcnZbKX5b76JvbF .cluster text{fill:#333;}#mermaid-svg-8lcnZbKX5b76JvbF .cluster span{color:#333;}#mermaid-svg-8lcnZbKX5b76JvbF div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8lcnZbKX5b76JvbF .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-8lcnZbKX5b76JvbF rect.text{fill:none;stroke-width:0;}#mermaid-svg-8lcnZbKX5b76JvbF .icon-shape,#mermaid-svg-8lcnZbKX5b76JvbF .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8lcnZbKX5b76JvbF .icon-shape p,#mermaid-svg-8lcnZbKX5b76JvbF .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-8lcnZbKX5b76JvbF .icon-shape .label rect,#mermaid-svg-8lcnZbKX5b76JvbF .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8lcnZbKX5b76JvbF .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-8lcnZbKX5b76JvbF .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-8lcnZbKX5b76JvbF :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

消息堆积

确认生产速度

确认消费速度

查看消费者错误日志

检查下游依赖是否慢

判断是否扩容消费者或降级处理

消息堆积时不要只扩消费者。要先确认消费者是不是因为下游慢、数据库慢、代码异常导致消费失败。如果根因不解决，扩容只会把压力更快打到下游。

项目难点怎么表达

很多技术面试会问“项目中遇到过什么难点”。这个问题不能回答成“我用了 Redis、MQ、ELK、Arthas”，工具名不是难点。

一个好的表达结构是：

#mermaid-svg-5dLabgc3QODyMVOq{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-5dLabgc3QODyMVOq .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-5dLabgc3QODyMVOq .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-5dLabgc3QODyMVOq .error-icon{fill:#552222;}#mermaid-svg-5dLabgc3QODyMVOq .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-5dLabgc3QODyMVOq .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-5dLabgc3QODyMVOq .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-5dLabgc3QODyMVOq .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-5dLabgc3QODyMVOq .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-5dLabgc3QODyMVOq .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-5dLabgc3QODyMVOq .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-5dLabgc3QODyMVOq .marker{fill:#333333;stroke:#333333;}#mermaid-svg-5dLabgc3QODyMVOq .marker.cross{stroke:#333333;}#mermaid-svg-5dLabgc3QODyMVOq svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-5dLabgc3QODyMVOq p{margin:0;}#mermaid-svg-5dLabgc3QODyMVOq .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-5dLabgc3QODyMVOq .cluster-label text{fill:#333;}#mermaid-svg-5dLabgc3QODyMVOq .cluster-label span{color:#333;}#mermaid-svg-5dLabgc3QODyMVOq .cluster-label span p{background-color:transparent;}#mermaid-svg-5dLabgc3QODyMVOq .label text,#mermaid-svg-5dLabgc3QODyMVOq span{fill:#333;color:#333;}#mermaid-svg-5dLabgc3QODyMVOq .node rect,#mermaid-svg-5dLabgc3QODyMVOq .node circle,#mermaid-svg-5dLabgc3QODyMVOq .node ellipse,#mermaid-svg-5dLabgc3QODyMVOq .node polygon,#mermaid-svg-5dLabgc3QODyMVOq .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-5dLabgc3QODyMVOq .rough-node .label text,#mermaid-svg-5dLabgc3QODyMVOq .node .label text,#mermaid-svg-5dLabgc3QODyMVOq .image-shape .label,#mermaid-svg-5dLabgc3QODyMVOq .icon-shape .label{text-anchor:middle;}#mermaid-svg-5dLabgc3QODyMVOq .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-5dLabgc3QODyMVOq .rough-node .label,#mermaid-svg-5dLabgc3QODyMVOq .node .label,#mermaid-svg-5dLabgc3QODyMVOq .image-shape .label,#mermaid-svg-5dLabgc3QODyMVOq .icon-shape .label{text-align:center;}#mermaid-svg-5dLabgc3QODyMVOq .node.clickable{cursor:pointer;}#mermaid-svg-5dLabgc3QODyMVOq .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-5dLabgc3QODyMVOq .arrowheadPath{fill:#333333;}#mermaid-svg-5dLabgc3QODyMVOq .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-5dLabgc3QODyMVOq .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-5dLabgc3QODyMVOq .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-5dLabgc3QODyMVOq .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-5dLabgc3QODyMVOq .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-5dLabgc3QODyMVOq .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-5dLabgc3QODyMVOq .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-5dLabgc3QODyMVOq .cluster text{fill:#333;}#mermaid-svg-5dLabgc3QODyMVOq .cluster span{color:#333;}#mermaid-svg-5dLabgc3QODyMVOq div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-5dLabgc3QODyMVOq .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-5dLabgc3QODyMVOq rect.text{fill:none;stroke-width:0;}#mermaid-svg-5dLabgc3QODyMVOq .icon-shape,#mermaid-svg-5dLabgc3QODyMVOq .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-5dLabgc3QODyMVOq .icon-shape p,#mermaid-svg-5dLabgc3QODyMVOq .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-5dLabgc3QODyMVOq .icon-shape .label rect,#mermaid-svg-5dLabgc3QODyMVOq .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-5dLabgc3QODyMVOq .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-5dLabgc3QODyMVOq .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-5dLabgc3QODyMVOq :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

背景

当时系统规模和业务目标

问题

具体故障或瓶颈表现

排查过程

用了哪些证据定位

解决方案

改了什么机制

结果

指标提升或故障减少

可以按这个模板讲：

当时订单服务在活动期间出现响应时间升高，主要表现是下单接口 P95 从 200ms 涨到 1.5s，错误率也有抬升。

我先通过监控确认不是全站故障，而是订单链路变慢；再用 traceId 查日志和链路追踪，发现耗时主要集中在库存扣减和优惠券锁定；随后结合数据库慢查询和连接池监控，确认优惠券服务存在慢 SQL，并且失败重试放大了数据库压力。

处理上，我们先临时降级非核心优惠券规则，降低活动期间的下游压力；随后优化 SQL 和索引，调整连接池与超时重试策略，并补充了慢调用告警。

最终下单接口 P95 恢复到 300ms 以内，错误率回落，后续活动期间没有再出现同类故障。

这段话有几个关键点：

这才像真实做过项目的人讲出来的。

一套生产排障清单

最后给一个实用清单。

#mermaid-svg-SdlMHOoNcAPOvNVf{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-SdlMHOoNcAPOvNVf .error-icon{fill:#552222;}#mermaid-svg-SdlMHOoNcAPOvNVf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-SdlMHOoNcAPOvNVf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-SdlMHOoNcAPOvNVf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-SdlMHOoNcAPOvNVf .marker.cross{stroke:#333333;}#mermaid-svg-SdlMHOoNcAPOvNVf svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-SdlMHOoNcAPOvNVf p{margin:0;}#mermaid-svg-SdlMHOoNcAPOvNVf .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf .cluster-label text{fill:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf .cluster-label span{color:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf .cluster-label span p{background-color:transparent;}#mermaid-svg-SdlMHOoNcAPOvNVf .label text,#mermaid-svg-SdlMHOoNcAPOvNVf span{fill:#333;color:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf .node rect,#mermaid-svg-SdlMHOoNcAPOvNVf .node circle,#mermaid-svg-SdlMHOoNcAPOvNVf .node ellipse,#mermaid-svg-SdlMHOoNcAPOvNVf .node polygon,#mermaid-svg-SdlMHOoNcAPOvNVf .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-SdlMHOoNcAPOvNVf .rough-node .label text,#mermaid-svg-SdlMHOoNcAPOvNVf .node .label text,#mermaid-svg-SdlMHOoNcAPOvNVf .image-shape .label,#mermaid-svg-SdlMHOoNcAPOvNVf .icon-shape .label{text-anchor:middle;}#mermaid-svg-SdlMHOoNcAPOvNVf .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-SdlMHOoNcAPOvNVf .rough-node .label,#mermaid-svg-SdlMHOoNcAPOvNVf .node .label,#mermaid-svg-SdlMHOoNcAPOvNVf .image-shape .label,#mermaid-svg-SdlMHOoNcAPOvNVf .icon-shape .label{text-align:center;}#mermaid-svg-SdlMHOoNcAPOvNVf .node.clickable{cursor:pointer;}#mermaid-svg-SdlMHOoNcAPOvNVf .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-SdlMHOoNcAPOvNVf .arrowheadPath{fill:#333333;}#mermaid-svg-SdlMHOoNcAPOvNVf .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-SdlMHOoNcAPOvNVf .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-SdlMHOoNcAPOvNVf .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SdlMHOoNcAPOvNVf .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-SdlMHOoNcAPOvNVf .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SdlMHOoNcAPOvNVf .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-SdlMHOoNcAPOvNVf .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-SdlMHOoNcAPOvNVf .cluster text{fill:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf .cluster span{color:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-SdlMHOoNcAPOvNVf .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-SdlMHOoNcAPOvNVf rect.text{fill:none;stroke-width:0;}#mermaid-svg-SdlMHOoNcAPOvNVf .icon-shape,#mermaid-svg-SdlMHOoNcAPOvNVf .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SdlMHOoNcAPOvNVf .icon-shape p,#mermaid-svg-SdlMHOoNcAPOvNVf .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-SdlMHOoNcAPOvNVf .icon-shape .label rect,#mermaid-svg-SdlMHOoNcAPOvNVf .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SdlMHOoNcAPOvNVf .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-SdlMHOoNcAPOvNVf .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-SdlMHOoNcAPOvNVf :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

告警出现

确认时间和影响范围

查看近期发布和配置变更

查询日志和 traceId

查看监控指标

查看链路追踪

定位瓶颈层级

必要时使用 Arthas

制定止血方案

验证指标恢复

补充长期修复和复盘

具体动作可以按优先级来：

生产排障的关键，不是会背多少工具，而是能不能沿着证据链把问题一步步压缩。

从“用户说慢”压缩到“某个接口慢”。
从“某个接口慢”压缩到“某个服务慢”。
从“某个服务慢”压缩到“某个方法、某条 SQL、某个下游调用慢”。
再从这个点拿出修复方案和验证结果。

这就是排障能力。