Apache Storm核心原理与实战场景全解析：实时流处理的“拓荒者“

Apache Storm核心原理与实战场景全解析：实时流处理的"拓荒者"

• • 前言
  • 一、什么是Apache Storm？
  • • 1.1 官方定义
    • 1.2 核心特质
    • 1.3 Storm在流处理生态中的定位
  • 二、Storm核心概念详解
  • • 2.1 元组（Tuple）
    • 2.2 流（Stream）
    • 2.3 拓扑（Topology）
    • 2.4 Spout（数据源）
    • 2.5 Bolt（处理逻辑）
    • 2.6 流分组（Stream Groupings）
  • 三、Storm系统架构
  • • 3.1 架构组件
    • 3.2 任务分配与执行流程
    • 3.3 容错机制
    • 3.4 数据可靠性保障
  • 四、核心应用场景
  • • 4.1 实时数据分析
    • 4.2 实时监控与预警
    • 4.3 实时推荐系统
    • 4.4 分布式ETL
    • 4.5 持续计算/机器学习
    • 4.6 金融风控
  • 五、Storm与Kafka的黄金组合
  • • 5.1 架构模式
    • 5.2 组合优势
    • 5.3 配置示例
  • 六、Storm的优缺点分析
  • • 6.1 核心优势
    • 6.2 局限性
    • 6.3 与其他框架对比
  • 七、如何选择技术栈？
  • • 7.1 推荐使用Storm的场景
    • 7.2 不推荐使用Storm的场景
    • 7.3 选型决策树
  • 八、实战：构建一个实时日志分析系统
  • • 8.1 需求描述
    • 8.2 拓扑设计
    • 8.3 核心代码实现
    • 8.4 部署与监控
  • 总结

前言

在实时计算领域，Apache Storm是一个里程碑式的存在。它像一位经验丰富的"急诊科医生"，能够在数据产生的毫秒级时间内做出反应和处理。自2011年开源以来，Storm以其极低的延迟、可靠的容错机制和简洁的编程模型，成为实时流处理领域的开拓者和经典实现。

本文将深入剖析Storm的核心概念、系统架构、工作原理，并通过详尽的流程图和代码示例，帮助读者全面理解这个实时计算框架的精髓。同时，我们也将探讨Storm的主要应用场景，以及在当前流处理框架百花齐放的时代，如何理性选择技术栈。

一、什么是Apache Storm？

1.1 官方定义

Apache Storm是一个分布式、可靠、容错的实时流式数据处理系统。它能够无界地处理源源不断产生的数据，让"实时计算"从理想走向现实。

1.2 核心特质

1.3 Storm在流处理生态中的定位

#mermaid-svg-WqiRAHP5918c2eME{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-WqiRAHP5918c2eME .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-WqiRAHP5918c2eME .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-WqiRAHP5918c2eME .error-icon{fill:#552222;}#mermaid-svg-WqiRAHP5918c2eME .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-WqiRAHP5918c2eME .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-WqiRAHP5918c2eME .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-WqiRAHP5918c2eME .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-WqiRAHP5918c2eME .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-WqiRAHP5918c2eME .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-WqiRAHP5918c2eME .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-WqiRAHP5918c2eME .marker{fill:#333333;stroke:#333333;}#mermaid-svg-WqiRAHP5918c2eME .marker.cross{stroke:#333333;}#mermaid-svg-WqiRAHP5918c2eME svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-WqiRAHP5918c2eME p{margin:0;}#mermaid-svg-WqiRAHP5918c2eME .edge{stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .section–1 rect,#mermaid-svg-WqiRAHP5918c2eME .section–1 path,#mermaid-svg-WqiRAHP5918c2eME .section–1 circle,#mermaid-svg-WqiRAHP5918c2eME .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section–1 text{fill:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth–1{stroke-width:17;}#mermaid-svg-WqiRAHP5918c2eME .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-0 rect,#mermaid-svg-WqiRAHP5918c2eME .section-0 path,#mermaid-svg-WqiRAHP5918c2eME .section-0 circle,#mermaid-svg-WqiRAHP5918c2eME .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-WqiRAHP5918c2eME .section-0 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-0{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-0{stroke-width:14;}#mermaid-svg-WqiRAHP5918c2eME .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-1 rect,#mermaid-svg-WqiRAHP5918c2eME .section-1 path,#mermaid-svg-WqiRAHP5918c2eME .section-1 circle,#mermaid-svg-WqiRAHP5918c2eME .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-1 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-1{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-1{stroke-width:11;}#mermaid-svg-WqiRAHP5918c2eME .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-2 rect,#mermaid-svg-WqiRAHP5918c2eME .section-2 path,#mermaid-svg-WqiRAHP5918c2eME .section-2 circle,#mermaid-svg-WqiRAHP5918c2eME .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-2 text{fill:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-2{stroke-width:8;}#mermaid-svg-WqiRAHP5918c2eME .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-3 rect,#mermaid-svg-WqiRAHP5918c2eME .section-3 path,#mermaid-svg-WqiRAHP5918c2eME .section-3 circle,#mermaid-svg-WqiRAHP5918c2eME .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-3 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-3{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-3{stroke-width:5;}#mermaid-svg-WqiRAHP5918c2eME .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-4 rect,#mermaid-svg-WqiRAHP5918c2eME .section-4 path,#mermaid-svg-WqiRAHP5918c2eME .section-4 circle,#mermaid-svg-WqiRAHP5918c2eME .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-4 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-4{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-4{stroke-width:2;}#mermaid-svg-WqiRAHP5918c2eME .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-5 rect,#mermaid-svg-WqiRAHP5918c2eME .section-5 path,#mermaid-svg-WqiRAHP5918c2eME .section-5 circle,#mermaid-svg-WqiRAHP5918c2eME .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-5 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-5{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-5{stroke-width:-1;}#mermaid-svg-WqiRAHP5918c2eME .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-6 rect,#mermaid-svg-WqiRAHP5918c2eME .section-6 path,#mermaid-svg-WqiRAHP5918c2eME .section-6 circle,#mermaid-svg-WqiRAHP5918c2eME .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-6 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-6{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-6{stroke-width:-4;}#mermaid-svg-WqiRAHP5918c2eME .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-7 rect,#mermaid-svg-WqiRAHP5918c2eME .section-7 path,#mermaid-svg-WqiRAHP5918c2eME .section-7 circle,#mermaid-svg-WqiRAHP5918c2eME .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-7 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-7{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-7{stroke-width:-7;}#mermaid-svg-WqiRAHP5918c2eME .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-8 rect,#mermaid-svg-WqiRAHP5918c2eME .section-8 path,#mermaid-svg-WqiRAHP5918c2eME .section-8 circle,#mermaid-svg-WqiRAHP5918c2eME .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-8 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-8{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-8{stroke-width:-10;}#mermaid-svg-WqiRAHP5918c2eME .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-9 rect,#mermaid-svg-WqiRAHP5918c2eME .section-9 path,#mermaid-svg-WqiRAHP5918c2eME .section-9 circle,#mermaid-svg-WqiRAHP5918c2eME .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-9 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-9{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-9{stroke-width:-13;}#mermaid-svg-WqiRAHP5918c2eME .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-10 rect,#mermaid-svg-WqiRAHP5918c2eME .section-10 path,#mermaid-svg-WqiRAHP5918c2eME .section-10 circle,#mermaid-svg-WqiRAHP5918c2eME .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-10 text{fill:black;}#mermaid-svg-WqiRAHP5918c2eME .node-icon-10{font-size:40px;color:black;}#mermaid-svg-WqiRAHP5918c2eME .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .edge-depth-10{stroke-width:-16;}#mermaid-svg-WqiRAHP5918c2eME .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-WqiRAHP5918c2eME .lineWrapper line{stroke:black;}#mermaid-svg-WqiRAHP5918c2eME .disabled,#mermaid-svg-WqiRAHP5918c2eME .disabled circle,#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:lightgray;}#mermaid-svg-WqiRAHP5918c2eME .disabled text{fill:#efefef;}#mermaid-svg-WqiRAHP5918c2eME .section-root rect,#mermaid-svg-WqiRAHP5918c2eME .section-root path,#mermaid-svg-WqiRAHP5918c2eME .section-root circle{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-WqiRAHP5918c2eME .section-root text{fill:#ffffff;}#mermaid-svg-WqiRAHP5918c2eME .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-WqiRAHP5918c2eME .edge{fill:none;}#mermaid-svg-WqiRAHP5918c2eME .eventWrapper{filter:brightness(120%);}#mermaid-svg-WqiRAHP5918c2eME :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

2011年

Storm诞生

开创实时流处理时代

2013年

Spark Streaming

微批处理架构

2014年

Flink

新一代流处理框架

2016年至今

Kafka Streams

轻量级库

多种框架并存

流处理框架演进史

Storm是最早的流式处理框架之一，技术成熟可靠。它奠定了流处理的基础概念——DAG拓扑、流、元组、Spout/Bolt等，后续的Flink、Spark Streaming等框架或多或少都借鉴了这些思想。

二、Storm核心概念详解

2.1 元组（Tuple）

Tuple是Storm核心数据结构，是消息传递的基本单元。它是一个不可变的Key-Value对，可以包含不同类型的数据。

// Tuple示例
// 声明输出字段
declareOutputFields(new Fields("word", "count"));

// 发射Tuple
collector.emit(new Values("hello", 1));

2.2 流（Stream）

Stream是Storm的关键抽象，是一个无边界的连续Tuple序列。流由Spout产生，被Bolt消费和处理。

2.3 拓扑（Topology）

Topology是整个实时应用程序的逻辑封装，由各个组件（Spout/Bolt）组成的一个有向无环图（DAG）。与Hadoop的MapReduce Job不同，Topology是一个长驻程序，一旦提交就会一直运行，除非人工中止。

#mermaid-svg-80DMy3QGs59RtMU3{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-80DMy3QGs59RtMU3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-80DMy3QGs59RtMU3 .error-icon{fill:#552222;}#mermaid-svg-80DMy3QGs59RtMU3 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-80DMy3QGs59RtMU3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-80DMy3QGs59RtMU3 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-80DMy3QGs59RtMU3 .marker.cross{stroke:#333333;}#mermaid-svg-80DMy3QGs59RtMU3 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-80DMy3QGs59RtMU3 p{margin:0;}#mermaid-svg-80DMy3QGs59RtMU3 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-80DMy3QGs59RtMU3 .cluster-label text{fill:#333;}#mermaid-svg-80DMy3QGs59RtMU3 .cluster-label span{color:#333;}#mermaid-svg-80DMy3QGs59RtMU3 .cluster-label span p{background-color:transparent;}#mermaid-svg-80DMy3QGs59RtMU3 .label text,#mermaid-svg-80DMy3QGs59RtMU3 span{fill:#333;color:#333;}#mermaid-svg-80DMy3QGs59RtMU3 .node rect,#mermaid-svg-80DMy3QGs59RtMU3 .node circle,#mermaid-svg-80DMy3QGs59RtMU3 .node ellipse,#mermaid-svg-80DMy3QGs59RtMU3 .node polygon,#mermaid-svg-80DMy3QGs59RtMU3 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-80DMy3QGs59RtMU3 .rough-node .label text,#mermaid-svg-80DMy3QGs59RtMU3 .node .label text,#mermaid-svg-80DMy3QGs59RtMU3 .image-shape .label,#mermaid-svg-80DMy3QGs59RtMU3 .icon-shape .label{text-anchor:middle;}#mermaid-svg-80DMy3QGs59RtMU3 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-80DMy3QGs59RtMU3 .rough-node .label,#mermaid-svg-80DMy3QGs59RtMU3 .node .label,#mermaid-svg-80DMy3QGs59RtMU3 .image-shape .label,#mermaid-svg-80DMy3QGs59RtMU3 .icon-shape .label{text-align:center;}#mermaid-svg-80DMy3QGs59RtMU3 .node.clickable{cursor:pointer;}#mermaid-svg-80DMy3QGs59RtMU3 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-80DMy3QGs59RtMU3 .arrowheadPath{fill:#333333;}#mermaid-svg-80DMy3QGs59RtMU3 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-80DMy3QGs59RtMU3 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-80DMy3QGs59RtMU3 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-80DMy3QGs59RtMU3 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-80DMy3QGs59RtMU3 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-80DMy3QGs59RtMU3 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-80DMy3QGs59RtMU3 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-80DMy3QGs59RtMU3 .cluster text{fill:#333;}#mermaid-svg-80DMy3QGs59RtMU3 .cluster span{color:#333;}#mermaid-svg-80DMy3QGs59RtMU3 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-80DMy3QGs59RtMU3 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-80DMy3QGs59RtMU3 rect.text{fill:none;stroke-width:0;}#mermaid-svg-80DMy3QGs59RtMU3 .icon-shape,#mermaid-svg-80DMy3QGs59RtMU3 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-80DMy3QGs59RtMU3 .icon-shape p,#mermaid-svg-80DMy3QGs59RtMU3 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-80DMy3QGs59RtMU3 .icon-shape rect,#mermaid-svg-80DMy3QGs59RtMU3 .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-80DMy3QGs59RtMU3 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-80DMy3QGs59RtMU3 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-80DMy3QGs59RtMU3 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Topology 示例：实时词频统计

Shuffle Grouping

Fields Grouping按单词分组

Spout数据源

Split Bolt分词

Count Bolt计数

Sink Bolt结果输出

2.4 Spout（数据源）

Spout是Topology中产生源数据的组件。它负责从外部数据源读取数据，并转换为Tuple发射到拓扑中。

常见的Spout实现：

• KafkaSpout：从Kafka消费数据
• RedisSpout：从Redis读取数据
• TwitterSpout：从Twitter API获取实时推文

2.5 Bolt（处理逻辑）

Bolt是Topology中接收数据并执行具体处理逻辑的组件。它可以完成过滤、统计、转换、聚合、持久化等操作。

// 自定义Bolt示例
public class WordCountBolt extends BaseRichBolt {
private OutputCollector collector;
private Map<String, Integer> counts;

@Override
public void prepare(Map conf, TopologyContext context,
OutputCollector collector) {
this.collector = collector;
this.counts = new HashMap<>();
}

@Override
public void execute(Tuple tuple) {
String word = tuple.getStringByField("word");
Integer count = counts.getOrDefault(word, 0) + 1;
counts.put(word, count);

// 发射处理结果
collector.emit(new Values(word, count));
// 确认处理完成
collector.ack(tuple);
}

@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word", "count"));
}
}

2.6 流分组（Stream Groupings）

Stream Groupings定义了Tuple在任务之间的分发策略。Storm内置了多种分组方式：

三、Storm系统架构

3.1 架构组件

Storm集群采用主从架构，主要由以下组件构成：

#mermaid-svg-p9FCcxUzMjamtc0Q{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-p9FCcxUzMjamtc0Q .error-icon{fill:#552222;}#mermaid-svg-p9FCcxUzMjamtc0Q .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-p9FCcxUzMjamtc0Q .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-p9FCcxUzMjamtc0Q .marker{fill:#333333;stroke:#333333;}#mermaid-svg-p9FCcxUzMjamtc0Q .marker.cross{stroke:#333333;}#mermaid-svg-p9FCcxUzMjamtc0Q svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-p9FCcxUzMjamtc0Q p{margin:0;}#mermaid-svg-p9FCcxUzMjamtc0Q .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q .cluster-label text{fill:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q .cluster-label span{color:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q .cluster-label span p{background-color:transparent;}#mermaid-svg-p9FCcxUzMjamtc0Q .label text,#mermaid-svg-p9FCcxUzMjamtc0Q span{fill:#333;color:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q .node rect,#mermaid-svg-p9FCcxUzMjamtc0Q .node circle,#mermaid-svg-p9FCcxUzMjamtc0Q .node ellipse,#mermaid-svg-p9FCcxUzMjamtc0Q .node polygon,#mermaid-svg-p9FCcxUzMjamtc0Q .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-p9FCcxUzMjamtc0Q .rough-node .label text,#mermaid-svg-p9FCcxUzMjamtc0Q .node .label text,#mermaid-svg-p9FCcxUzMjamtc0Q .image-shape .label,#mermaid-svg-p9FCcxUzMjamtc0Q .icon-shape .label{text-anchor:middle;}#mermaid-svg-p9FCcxUzMjamtc0Q .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-p9FCcxUzMjamtc0Q .rough-node .label,#mermaid-svg-p9FCcxUzMjamtc0Q .node .label,#mermaid-svg-p9FCcxUzMjamtc0Q .image-shape .label,#mermaid-svg-p9FCcxUzMjamtc0Q .icon-shape .label{text-align:center;}#mermaid-svg-p9FCcxUzMjamtc0Q .node.clickable{cursor:pointer;}#mermaid-svg-p9FCcxUzMjamtc0Q .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-p9FCcxUzMjamtc0Q .arrowheadPath{fill:#333333;}#mermaid-svg-p9FCcxUzMjamtc0Q .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-p9FCcxUzMjamtc0Q .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-p9FCcxUzMjamtc0Q .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-p9FCcxUzMjamtc0Q .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-p9FCcxUzMjamtc0Q .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-p9FCcxUzMjamtc0Q .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-p9FCcxUzMjamtc0Q .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-p9FCcxUzMjamtc0Q .cluster text{fill:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q .cluster span{color:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-p9FCcxUzMjamtc0Q .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-p9FCcxUzMjamtc0Q rect.text{fill:none;stroke-width:0;}#mermaid-svg-p9FCcxUzMjamtc0Q .icon-shape,#mermaid-svg-p9FCcxUzMjamtc0Q .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-p9FCcxUzMjamtc0Q .icon-shape p,#mermaid-svg-p9FCcxUzMjamtc0Q .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-p9FCcxUzMjamtc0Q .icon-shape rect,#mermaid-svg-p9FCcxUzMjamtc0Q .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-p9FCcxUzMjamtc0Q .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-p9FCcxUzMjamtc0Q .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-p9FCcxUzMjamtc0Q :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Storm 集群

从节点2

从节点1

提交拓扑

Nimbus主节点

ZooKeeper协调服务

Supervisor

Worker进程

Worker进程

Supervisor

Worker进程

Worker进程

Storm UI监控界面

客户端

3.2 任务分配与执行流程

Worker

Supervisor

ZooKeeper

Nimbus

客户端

Worker

Supervisor

ZooKeeper

Nimbus

客户端

#mermaid-svg-Bmx1DCyO16liYoOP{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Bmx1DCyO16liYoOP .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Bmx1DCyO16liYoOP .error-icon{fill:#552222;}#mermaid-svg-Bmx1DCyO16liYoOP .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Bmx1DCyO16liYoOP .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Bmx1DCyO16liYoOP .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Bmx1DCyO16liYoOP .marker.cross{stroke:#333333;}#mermaid-svg-Bmx1DCyO16liYoOP svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Bmx1DCyO16liYoOP p{margin:0;}#mermaid-svg-Bmx1DCyO16liYoOP .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-Bmx1DCyO16liYoOP text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-Bmx1DCyO16liYoOP .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-Bmx1DCyO16liYoOP .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-Bmx1DCyO16liYoOP .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-Bmx1DCyO16liYoOP .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-Bmx1DCyO16liYoOP #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-Bmx1DCyO16liYoOP .sequenceNumber{fill:white;}#mermaid-svg-Bmx1DCyO16liYoOP #sequencenumber{fill:#333;}#mermaid-svg-Bmx1DCyO16liYoOP #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-Bmx1DCyO16liYoOP .messageText{fill:#333;stroke:none;}#mermaid-svg-Bmx1DCyO16liYoOP .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-Bmx1DCyO16liYoOP .labelText,#mermaid-svg-Bmx1DCyO16liYoOP .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-Bmx1DCyO16liYoOP .loopText,#mermaid-svg-Bmx1DCyO16liYoOP .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-Bmx1DCyO16liYoOP .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-Bmx1DCyO16liYoOP .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-Bmx1DCyO16liYoOP .noteText,#mermaid-svg-Bmx1DCyO16liYoOP .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-Bmx1DCyO16liYoOP .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-Bmx1DCyO16liYoOP .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-Bmx1DCyO16liYoOP .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-Bmx1DCyO16liYoOP .actorPopupMenu{position:absolute;}#mermaid-svg-Bmx1DCyO16liYoOP .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-Bmx1DCyO16liYoOP .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-Bmx1DCyO16liYoOP .actor-man circle,#mermaid-svg-Bmx1DCyO16liYoOP line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-Bmx1DCyO16liYoOP :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

1. 提交拓扑

2. 解析拓扑、计算分配

3. 写入任务分配信息

4. 监听到任务变化

5. 拉起Worker进程

6. 分配具体任务

7. 启动Executor/Task

8. 上报心跳

3.3 容错机制

Storm的容错能力体现在多个层面：

3.4 数据可靠性保障

Storm提供三种级别的数据可靠性：

#mermaid-svg-MQtkLqkMpkmtY74l{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-MQtkLqkMpkmtY74l .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-MQtkLqkMpkmtY74l .error-icon{fill:#552222;}#mermaid-svg-MQtkLqkMpkmtY74l .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-MQtkLqkMpkmtY74l .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-MQtkLqkMpkmtY74l .marker{fill:#333333;stroke:#333333;}#mermaid-svg-MQtkLqkMpkmtY74l .marker.cross{stroke:#333333;}#mermaid-svg-MQtkLqkMpkmtY74l svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-MQtkLqkMpkmtY74l p{margin:0;}#mermaid-svg-MQtkLqkMpkmtY74l .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-MQtkLqkMpkmtY74l .cluster-label text{fill:#333;}#mermaid-svg-MQtkLqkMpkmtY74l .cluster-label span{color:#333;}#mermaid-svg-MQtkLqkMpkmtY74l .cluster-label span p{background-color:transparent;}#mermaid-svg-MQtkLqkMpkmtY74l .label text,#mermaid-svg-MQtkLqkMpkmtY74l span{fill:#333;color:#333;}#mermaid-svg-MQtkLqkMpkmtY74l .node rect,#mermaid-svg-MQtkLqkMpkmtY74l .node circle,#mermaid-svg-MQtkLqkMpkmtY74l .node ellipse,#mermaid-svg-MQtkLqkMpkmtY74l .node polygon,#mermaid-svg-MQtkLqkMpkmtY74l .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-MQtkLqkMpkmtY74l .rough-node .label text,#mermaid-svg-MQtkLqkMpkmtY74l .node .label text,#mermaid-svg-MQtkLqkMpkmtY74l .image-shape .label,#mermaid-svg-MQtkLqkMpkmtY74l .icon-shape .label{text-anchor:middle;}#mermaid-svg-MQtkLqkMpkmtY74l .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-MQtkLqkMpkmtY74l .rough-node .label,#mermaid-svg-MQtkLqkMpkmtY74l .node .label,#mermaid-svg-MQtkLqkMpkmtY74l .image-shape .label,#mermaid-svg-MQtkLqkMpkmtY74l .icon-shape .label{text-align:center;}#mermaid-svg-MQtkLqkMpkmtY74l .node.clickable{cursor:pointer;}#mermaid-svg-MQtkLqkMpkmtY74l .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-MQtkLqkMpkmtY74l .arrowheadPath{fill:#333333;}#mermaid-svg-MQtkLqkMpkmtY74l .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-MQtkLqkMpkmtY74l .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-MQtkLqkMpkmtY74l .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MQtkLqkMpkmtY74l .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-MQtkLqkMpkmtY74l .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MQtkLqkMpkmtY74l .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-MQtkLqkMpkmtY74l .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-MQtkLqkMpkmtY74l .cluster text{fill:#333;}#mermaid-svg-MQtkLqkMpkmtY74l .cluster span{color:#333;}#mermaid-svg-MQtkLqkMpkmtY74l div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-MQtkLqkMpkmtY74l .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-MQtkLqkMpkmtY74l rect.text{fill:none;stroke-width:0;}#mermaid-svg-MQtkLqkMpkmtY74l .icon-shape,#mermaid-svg-MQtkLqkMpkmtY74l .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MQtkLqkMpkmtY74l .icon-shape p,#mermaid-svg-MQtkLqkMpkmtY74l .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-MQtkLqkMpkmtY74l .icon-shape rect,#mermaid-svg-MQtkLqkMpkmtY74l .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MQtkLqkMpkmtY74l .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-MQtkLqkMpkmtY74l .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-MQtkLqkMpkmtY74l :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

特点

可靠性级别

至多一次At-most-once

至少一次At-least-once

精确一次Exactly-once

可能丢数据性能最高

可能重复处理性能中等

不丢不重性能较低

Acker机制是实现可靠性的核心：通过异或运算跟踪Tuple树，确保每个元组被完整处理或超时重发。

四、核心应用场景

4.1 实时数据分析

Storm毫秒级的延迟使其成为实时数据分析的理想选择。

#mermaid-svg-0JxlZBgTsmDJDXBP{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-0JxlZBgTsmDJDXBP .error-icon{fill:#552222;}#mermaid-svg-0JxlZBgTsmDJDXBP .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-0JxlZBgTsmDJDXBP .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-0JxlZBgTsmDJDXBP .marker{fill:#333333;stroke:#333333;}#mermaid-svg-0JxlZBgTsmDJDXBP .marker.cross{stroke:#333333;}#mermaid-svg-0JxlZBgTsmDJDXBP svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-0JxlZBgTsmDJDXBP p{margin:0;}#mermaid-svg-0JxlZBgTsmDJDXBP .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP .cluster-label text{fill:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP .cluster-label span{color:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP .cluster-label span p{background-color:transparent;}#mermaid-svg-0JxlZBgTsmDJDXBP .label text,#mermaid-svg-0JxlZBgTsmDJDXBP span{fill:#333;color:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP .node rect,#mermaid-svg-0JxlZBgTsmDJDXBP .node circle,#mermaid-svg-0JxlZBgTsmDJDXBP .node ellipse,#mermaid-svg-0JxlZBgTsmDJDXBP .node polygon,#mermaid-svg-0JxlZBgTsmDJDXBP .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-0JxlZBgTsmDJDXBP .rough-node .label text,#mermaid-svg-0JxlZBgTsmDJDXBP .node .label text,#mermaid-svg-0JxlZBgTsmDJDXBP .image-shape .label,#mermaid-svg-0JxlZBgTsmDJDXBP .icon-shape .label{text-anchor:middle;}#mermaid-svg-0JxlZBgTsmDJDXBP .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-0JxlZBgTsmDJDXBP .rough-node .label,#mermaid-svg-0JxlZBgTsmDJDXBP .node .label,#mermaid-svg-0JxlZBgTsmDJDXBP .image-shape .label,#mermaid-svg-0JxlZBgTsmDJDXBP .icon-shape .label{text-align:center;}#mermaid-svg-0JxlZBgTsmDJDXBP .node.clickable{cursor:pointer;}#mermaid-svg-0JxlZBgTsmDJDXBP .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-0JxlZBgTsmDJDXBP .arrowheadPath{fill:#333333;}#mermaid-svg-0JxlZBgTsmDJDXBP .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-0JxlZBgTsmDJDXBP .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-0JxlZBgTsmDJDXBP .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0JxlZBgTsmDJDXBP .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-0JxlZBgTsmDJDXBP .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0JxlZBgTsmDJDXBP .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-0JxlZBgTsmDJDXBP .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-0JxlZBgTsmDJDXBP .cluster text{fill:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP .cluster span{color:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-0JxlZBgTsmDJDXBP .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-0JxlZBgTsmDJDXBP rect.text{fill:none;stroke-width:0;}#mermaid-svg-0JxlZBgTsmDJDXBP .icon-shape,#mermaid-svg-0JxlZBgTsmDJDXBP .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0JxlZBgTsmDJDXBP .icon-shape p,#mermaid-svg-0JxlZBgTsmDJDXBP .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-0JxlZBgTsmDJDXBP .icon-shape rect,#mermaid-svg-0JxlZBgTsmDJDXBP .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0JxlZBgTsmDJDXBP .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-0JxlZBgTsmDJDXBP .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-0JxlZBgTsmDJDXBP :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

实时用户行为分析

Kafka

实时统计

持久化

用户行为日志

Storm拓扑

Redis缓存

数据库

实时看板

离线报表

典型案例：电商网站的实时用户行为分析，包括页面浏览、点击、加购、下单等事件的实时统计。

4.2 实时监控与预警

在金融、运维等领域，实时监控和预警至关重要。

4.3 实时推荐系统

// 实时推荐Topology骨架
TopologyBuilder builder = new TopologyBuilder();

// 从Kafka读取用户行为
builder.setSpout("click-stream", new KafkaSpout(kafkaConfig), 5);

// 实时特征提取
builder.setBolt("feature-extract", new FeatureExtractBolt(), 10)
.shuffleGrouping("click-stream");

// 调用推荐模型
builder.setBolt("recommend", new RecommendBolt(), 20)
.fieldsGrouping("feature-extract", new Fields("userId"));

// 结果输出
builder.setBolt("output", new OutputBolt(), 5)
.shuffleGrouping("recommend");

4.4 分布式ETL

Storm可用于实时数据清洗、转换和加载：

#mermaid-svg-KUoH55A7LtSVt9bo{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-KUoH55A7LtSVt9bo .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-KUoH55A7LtSVt9bo .error-icon{fill:#552222;}#mermaid-svg-KUoH55A7LtSVt9bo .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-KUoH55A7LtSVt9bo .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-KUoH55A7LtSVt9bo .marker{fill:#333333;stroke:#333333;}#mermaid-svg-KUoH55A7LtSVt9bo .marker.cross{stroke:#333333;}#mermaid-svg-KUoH55A7LtSVt9bo svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-KUoH55A7LtSVt9bo p{margin:0;}#mermaid-svg-KUoH55A7LtSVt9bo .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-KUoH55A7LtSVt9bo .cluster-label text{fill:#333;}#mermaid-svg-KUoH55A7LtSVt9bo .cluster-label span{color:#333;}#mermaid-svg-KUoH55A7LtSVt9bo .cluster-label span p{background-color:transparent;}#mermaid-svg-KUoH55A7LtSVt9bo .label text,#mermaid-svg-KUoH55A7LtSVt9bo span{fill:#333;color:#333;}#mermaid-svg-KUoH55A7LtSVt9bo .node rect,#mermaid-svg-KUoH55A7LtSVt9bo .node circle,#mermaid-svg-KUoH55A7LtSVt9bo .node ellipse,#mermaid-svg-KUoH55A7LtSVt9bo .node polygon,#mermaid-svg-KUoH55A7LtSVt9bo .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-KUoH55A7LtSVt9bo .rough-node .label text,#mermaid-svg-KUoH55A7LtSVt9bo .node .label text,#mermaid-svg-KUoH55A7LtSVt9bo .image-shape .label,#mermaid-svg-KUoH55A7LtSVt9bo .icon-shape .label{text-anchor:middle;}#mermaid-svg-KUoH55A7LtSVt9bo .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-KUoH55A7LtSVt9bo .rough-node .label,#mermaid-svg-KUoH55A7LtSVt9bo .node .label,#mermaid-svg-KUoH55A7LtSVt9bo .image-shape .label,#mermaid-svg-KUoH55A7LtSVt9bo .icon-shape .label{text-align:center;}#mermaid-svg-KUoH55A7LtSVt9bo .node.clickable{cursor:pointer;}#mermaid-svg-KUoH55A7LtSVt9bo .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-KUoH55A7LtSVt9bo .arrowheadPath{fill:#333333;}#mermaid-svg-KUoH55A7LtSVt9bo .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-KUoH55A7LtSVt9bo .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-KUoH55A7LtSVt9bo .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KUoH55A7LtSVt9bo .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-KUoH55A7LtSVt9bo .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KUoH55A7LtSVt9bo .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-KUoH55A7LtSVt9bo .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-KUoH55A7LtSVt9bo .cluster text{fill:#333;}#mermaid-svg-KUoH55A7LtSVt9bo .cluster span{color:#333;}#mermaid-svg-KUoH55A7LtSVt9bo div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-KUoH55A7LtSVt9bo .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-KUoH55A7LtSVt9bo rect.text{fill:none;stroke-width:0;}#mermaid-svg-KUoH55A7LtSVt9bo .icon-shape,#mermaid-svg-KUoH55A7LtSVt9bo .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KUoH55A7LtSVt9bo .icon-shape p,#mermaid-svg-KUoH55A7LtSVt9bo .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-KUoH55A7LtSVt9bo .icon-shape rect,#mermaid-svg-KUoH55A7LtSVt9bo .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KUoH55A7LtSVt9bo .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-KUoH55A7LtSVt9bo .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-KUoH55A7LtSVt9bo :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Kafka

原始数据JSON格式

Parse Bolt解析

Filter Bolt过滤无效数据

Transform Bolt格式转换

Enrich Bolt关联维表

Sink Bolt写入HBase

4.5 持续计算/机器学习

Storm支持在线机器学习模型的实时预测：

# Python Bolt示例（使用pystorm）
from pystorm import Storm
import pickle

class MLPredictionBolt(Storm.BasicBolt):
def initialize(self, conf, context):
# 加载预训练模型
with open('model.pkl', 'rb') as f:
self.model = pickle.load(f)

def process(self, tup):
features = tup.values[0] # 输入特征
prediction = self.model.predict([features])[0]
self.emit([prediction])

4.6 金融风控

在金融领域，毫秒级的延迟意味着巨大的价值：

• 实时反欺诈：基于用户行为模式识别异常交易
• 信用评分：实时计算用户信用分变化
• 交易监控：识别可疑交易模式

五、Storm与Kafka的黄金组合

在实际生产中，Storm + Kafka是最经典的实时数据处理组合。

5.1 架构模式

#mermaid-svg-chAfahZSWU1eKVvy{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-chAfahZSWU1eKVvy .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-chAfahZSWU1eKVvy .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-chAfahZSWU1eKVvy .error-icon{fill:#552222;}#mermaid-svg-chAfahZSWU1eKVvy .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-chAfahZSWU1eKVvy .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-chAfahZSWU1eKVvy .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-chAfahZSWU1eKVvy .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-chAfahZSWU1eKVvy .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-chAfahZSWU1eKVvy .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-chAfahZSWU1eKVvy .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-chAfahZSWU1eKVvy .marker{fill:#333333;stroke:#333333;}#mermaid-svg-chAfahZSWU1eKVvy .marker.cross{stroke:#333333;}#mermaid-svg-chAfahZSWU1eKVvy svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-chAfahZSWU1eKVvy p{margin:0;}#mermaid-svg-chAfahZSWU1eKVvy .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-chAfahZSWU1eKVvy .cluster-label text{fill:#333;}#mermaid-svg-chAfahZSWU1eKVvy .cluster-label span{color:#333;}#mermaid-svg-chAfahZSWU1eKVvy .cluster-label span p{background-color:transparent;}#mermaid-svg-chAfahZSWU1eKVvy .label text,#mermaid-svg-chAfahZSWU1eKVvy span{fill:#333;color:#333;}#mermaid-svg-chAfahZSWU1eKVvy .node rect,#mermaid-svg-chAfahZSWU1eKVvy .node circle,#mermaid-svg-chAfahZSWU1eKVvy .node ellipse,#mermaid-svg-chAfahZSWU1eKVvy .node polygon,#mermaid-svg-chAfahZSWU1eKVvy .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-chAfahZSWU1eKVvy .rough-node .label text,#mermaid-svg-chAfahZSWU1eKVvy .node .label text,#mermaid-svg-chAfahZSWU1eKVvy .image-shape .label,#mermaid-svg-chAfahZSWU1eKVvy .icon-shape .label{text-anchor:middle;}#mermaid-svg-chAfahZSWU1eKVvy .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-chAfahZSWU1eKVvy .rough-node .label,#mermaid-svg-chAfahZSWU1eKVvy .node .label,#mermaid-svg-chAfahZSWU1eKVvy .image-shape .label,#mermaid-svg-chAfahZSWU1eKVvy .icon-shape .label{text-align:center;}#mermaid-svg-chAfahZSWU1eKVvy .node.clickable{cursor:pointer;}#mermaid-svg-chAfahZSWU1eKVvy .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-chAfahZSWU1eKVvy .arrowheadPath{fill:#333333;}#mermaid-svg-chAfahZSWU1eKVvy .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-chAfahZSWU1eKVvy .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-chAfahZSWU1eKVvy .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-chAfahZSWU1eKVvy .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-chAfahZSWU1eKVvy .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-chAfahZSWU1eKVvy .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-chAfahZSWU1eKVvy .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-chAfahZSWU1eKVvy .cluster text{fill:#333;}#mermaid-svg-chAfahZSWU1eKVvy .cluster span{color:#333;}#mermaid-svg-chAfahZSWU1eKVvy div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-chAfahZSWU1eKVvy .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-chAfahZSWU1eKVvy rect.text{fill:none;stroke-width:0;}#mermaid-svg-chAfahZSWU1eKVvy .icon-shape,#mermaid-svg-chAfahZSWU1eKVvy .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-chAfahZSWU1eKVvy .icon-shape p,#mermaid-svg-chAfahZSWU1eKVvy .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-chAfahZSWU1eKVvy .icon-shape rect,#mermaid-svg-chAfahZSWU1eKVvy .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-chAfahZSWU1eKVvy .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-chAfahZSWU1eKVvy .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-chAfahZSWU1eKVvy :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

存储层

计算层

数据源层

消息队列层

日志

Kafka集群数据缓冲

数据库CDC

传感器

Storm拓扑实时处理

HBase

Elasticsearch

MySQL

5.2 组合优势

5.3 配置示例

// Kafka Spout配置
public KafkaSpout<String, String> createKafkaSpout() {
// Kafka连接配置
KafkaSpoutConfig<String, String> config =
KafkaSpoutConfig.builder("localhost:9092", "input-topic")
.setGroupId("storm-consumer-group")
.setFirstPollOffsetStrategy(KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST)
.setOffsetCommitPeriodMs(10000)
.setMaxPollRecords(500)
.setProcessingGuarantee(KafkaSpoutConfig.ProcessingGuarantee.AT_LEAST_ONCE)
.setRecordTranslator((record) -> {
// 将Kafka记录转换为Storm Tuple
return new Values(record.topic(), record.partition(),
record.offset(), record.value());
}, new Fields("topic", "partition", "offset", "value"))
.build();

return new KafkaSpout<>(config);
}

六、Storm的优缺点分析

6.1 核心优势

6.2 局限性

6.3 与其他框架对比

七、如何选择技术栈？

7.1 推荐使用Storm的场景

• ✅ 超低延迟需求：如金融交易监控（<100ms）
• ✅ 简单ETL处理：日志清洗、数据归一化等轻量级任务
• ✅ 遗留系统维护：已有Storm集群的渐进式升级
• ✅ 事件驱动型简单用例：基于事件的触发式处理

7.2 不推荐使用Storm的场景

• ❌ 复杂状态处理：需多级聚合或状态回溯的业务
• ❌ 批流统一需求：需同时处理离线与实时数据的场景
• ❌ 高级窗口计算：复杂的滑动窗口、会话窗口分析
• ❌ 机器学习集成：需原生ML库支持的场景

7.3 选型决策树

#mermaid-svg-kkLAoCShjDOVXIwn{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-kkLAoCShjDOVXIwn .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-kkLAoCShjDOVXIwn .error-icon{fill:#552222;}#mermaid-svg-kkLAoCShjDOVXIwn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-kkLAoCShjDOVXIwn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-kkLAoCShjDOVXIwn .marker{fill:#333333;stroke:#333333;}#mermaid-svg-kkLAoCShjDOVXIwn .marker.cross{stroke:#333333;}#mermaid-svg-kkLAoCShjDOVXIwn svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-kkLAoCShjDOVXIwn p{margin:0;}#mermaid-svg-kkLAoCShjDOVXIwn .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-kkLAoCShjDOVXIwn .cluster-label text{fill:#333;}#mermaid-svg-kkLAoCShjDOVXIwn .cluster-label span{color:#333;}#mermaid-svg-kkLAoCShjDOVXIwn .cluster-label span p{background-color:transparent;}#mermaid-svg-kkLAoCShjDOVXIwn .label text,#mermaid-svg-kkLAoCShjDOVXIwn span{fill:#333;color:#333;}#mermaid-svg-kkLAoCShjDOVXIwn .node rect,#mermaid-svg-kkLAoCShjDOVXIwn .node circle,#mermaid-svg-kkLAoCShjDOVXIwn .node ellipse,#mermaid-svg-kkLAoCShjDOVXIwn .node polygon,#mermaid-svg-kkLAoCShjDOVXIwn .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-kkLAoCShjDOVXIwn .rough-node .label text,#mermaid-svg-kkLAoCShjDOVXIwn .node .label text,#mermaid-svg-kkLAoCShjDOVXIwn .image-shape .label,#mermaid-svg-kkLAoCShjDOVXIwn .icon-shape .label{text-anchor:middle;}#mermaid-svg-kkLAoCShjDOVXIwn .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-kkLAoCShjDOVXIwn .rough-node .label,#mermaid-svg-kkLAoCShjDOVXIwn .node .label,#mermaid-svg-kkLAoCShjDOVXIwn .image-shape .label,#mermaid-svg-kkLAoCShjDOVXIwn .icon-shape .label{text-align:center;}#mermaid-svg-kkLAoCShjDOVXIwn .node.clickable{cursor:pointer;}#mermaid-svg-kkLAoCShjDOVXIwn .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-kkLAoCShjDOVXIwn .arrowheadPath{fill:#333333;}#mermaid-svg-kkLAoCShjDOVXIwn .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-kkLAoCShjDOVXIwn .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-kkLAoCShjDOVXIwn .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-kkLAoCShjDOVXIwn .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-kkLAoCShjDOVXIwn .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-kkLAoCShjDOVXIwn .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-kkLAoCShjDOVXIwn .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-kkLAoCShjDOVXIwn .cluster text{fill:#333;}#mermaid-svg-kkLAoCShjDOVXIwn .cluster span{color:#333;}#mermaid-svg-kkLAoCShjDOVXIwn div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-kkLAoCShjDOVXIwn .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-kkLAoCShjDOVXIwn rect.text{fill:none;stroke-width:0;}#mermaid-svg-kkLAoCShjDOVXIwn .icon-shape,#mermaid-svg-kkLAoCShjDOVXIwn .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-kkLAoCShjDOVXIwn .icon-shape p,#mermaid-svg-kkLAoCShjDOVXIwn .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-kkLAoCShjDOVXIwn .icon-shape rect,#mermaid-svg-kkLAoCShjDOVXIwn .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-kkLAoCShjDOVXIwn .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-kkLAoCShjDOVXIwn .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-kkLAoCShjDOVXIwn :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

<100ms

秒级

简单或无状态

复杂状态

已有Kafka

已有Storm

新项目

开始选型

延迟要求?

状态管理复杂度?

Spark Streaming

技术栈现状?

Flink

Kafka Streams

Storm

八、实战：构建一个实时日志分析系统

8.1 需求描述

实时分析Web服务器日志，统计：

• 每分钟各API的访问量
• 实时检测5xx错误并告警
• 统计TOP10热门URL

8.2 拓扑设计

#mermaid-svg-teiCHSDxAUQlMHNt{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-teiCHSDxAUQlMHNt .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-teiCHSDxAUQlMHNt .error-icon{fill:#552222;}#mermaid-svg-teiCHSDxAUQlMHNt .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-teiCHSDxAUQlMHNt .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-teiCHSDxAUQlMHNt .marker{fill:#333333;stroke:#333333;}#mermaid-svg-teiCHSDxAUQlMHNt .marker.cross{stroke:#333333;}#mermaid-svg-teiCHSDxAUQlMHNt svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-teiCHSDxAUQlMHNt p{margin:0;}#mermaid-svg-teiCHSDxAUQlMHNt .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-teiCHSDxAUQlMHNt .cluster-label text{fill:#333;}#mermaid-svg-teiCHSDxAUQlMHNt .cluster-label span{color:#333;}#mermaid-svg-teiCHSDxAUQlMHNt .cluster-label span p{background-color:transparent;}#mermaid-svg-teiCHSDxAUQlMHNt .label text,#mermaid-svg-teiCHSDxAUQlMHNt span{fill:#333;color:#333;}#mermaid-svg-teiCHSDxAUQlMHNt .node rect,#mermaid-svg-teiCHSDxAUQlMHNt .node circle,#mermaid-svg-teiCHSDxAUQlMHNt .node ellipse,#mermaid-svg-teiCHSDxAUQlMHNt .node polygon,#mermaid-svg-teiCHSDxAUQlMHNt .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-teiCHSDxAUQlMHNt .rough-node .label text,#mermaid-svg-teiCHSDxAUQlMHNt .node .label text,#mermaid-svg-teiCHSDxAUQlMHNt .image-shape .label,#mermaid-svg-teiCHSDxAUQlMHNt .icon-shape .label{text-anchor:middle;}#mermaid-svg-teiCHSDxAUQlMHNt .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-teiCHSDxAUQlMHNt .rough-node .label,#mermaid-svg-teiCHSDxAUQlMHNt .node .label,#mermaid-svg-teiCHSDxAUQlMHNt .image-shape .label,#mermaid-svg-teiCHSDxAUQlMHNt .icon-shape .label{text-align:center;}#mermaid-svg-teiCHSDxAUQlMHNt .node.clickable{cursor:pointer;}#mermaid-svg-teiCHSDxAUQlMHNt .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-teiCHSDxAUQlMHNt .arrowheadPath{fill:#333333;}#mermaid-svg-teiCHSDxAUQlMHNt .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-teiCHSDxAUQlMHNt .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-teiCHSDxAUQlMHNt .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-teiCHSDxAUQlMHNt .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-teiCHSDxAUQlMHNt .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-teiCHSDxAUQlMHNt .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-teiCHSDxAUQlMHNt .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-teiCHSDxAUQlMHNt .cluster text{fill:#333;}#mermaid-svg-teiCHSDxAUQlMHNt .cluster span{color:#333;}#mermaid-svg-teiCHSDxAUQlMHNt div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-teiCHSDxAUQlMHNt .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-teiCHSDxAUQlMHNt rect.text{fill:none;stroke-width:0;}#mermaid-svg-teiCHSDxAUQlMHNt .icon-shape,#mermaid-svg-teiCHSDxAUQlMHNt .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-teiCHSDxAUQlMHNt .icon-shape p,#mermaid-svg-teiCHSDxAUQlMHNt .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-teiCHSDxAUQlMHNt .icon-shape rect,#mermaid-svg-teiCHSDxAUQlMHNt .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-teiCHSDxAUQlMHNt .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-teiCHSDxAUQlMHNt .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-teiCHSDxAUQlMHNt :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

实时日志分析拓扑

Shuffle

Fields: api

Fields: status

Fields: url

KafkaSpout读取日志

ParseBolt解析日志

AccessCountBoltAPI访问计数

ErrorDetectBolt错误检测

UrlRankBoltURL排名

RedisSinkBolt写入Redis

AlertBolt发送告警

HBaseSinkBolt持久化

8.3 核心代码实现

public class LogAnalysisTopology {
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();

// 1. Kafka数据源
builder.setSpout("kafka-spout",
createKafkaSpout("weblog-topic"), 5);

// 2. 日志解析
builder.setBolt("parse-bolt", new ParseLogBolt(), 10)
.shuffleGrouping("kafka-spout");

// 3. API访问计数（每分钟）
builder.setBolt("access-count", new AccessCountBolt()
.withWindow(CountWindow.of(1, TimeUnit.MINUTES)), 8)
.fieldsGrouping("parse-bolt", new Fields("api"));

// 4. 错误检测
builder.setBolt("error-detect", new ErrorDetectBolt(), 4)
.fieldsGrouping("parse-bolt", new Fields("status"));

// 5. TOP URL统计
builder.setBolt("url-rank", new UrlRankBolt(), 6)
.fieldsGrouping("parse-bolt", new Fields("url"));

// 6. 结果输出
builder.setBolt("redis-sink", new RedisSinkBolt(), 5)
.shuffleGrouping("access-count");
builder.setBolt("alert-sink", new AlertBolt(), 2)
.shuffleGrouping("error-detect");
builder.setBolt("hbase-sink", new HBaseSinkBolt(), 5)
.shuffleGrouping("url-rank");

// 提交拓扑
Config conf = new Config();
conf.setDebug(false);
conf.setNumWorkers(10);

if (args.length > 0) {
// 集群模式
StormSubmitter.submitTopology(args[0], conf,
builder.createTopology());
} else {
// 本地模式
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("log-analysis", conf,
builder.createTopology());
}
}
}

8.4 部署与监控

# 打包拓扑
mvn clean package

# 提交到Storm集群
storm jar target/log-analysis-1.0.jar \\
com.example.LogAnalysisTopology \\
log-analysis-topology

# 查看拓扑状态
storm list
# 访问Storm UI: http://nimbus-host:8080

总结

Apache Storm作为实时流处理的"拓荒者"，以其极低延迟、高可靠、简洁模型的特性，在实时计算领域占据着重要地位。虽然随着Flink、Spark Streaming等新框架的崛起，Storm在一些复杂场景下显得力不从心，但在简单实时处理、超低延迟需求、事件驱动型应用等领域，Storm仍然是可靠的选择。

核心要点回顾：

理解Storm的设计思想和应用场景，不仅有助于我们正确使用这一工具，更能帮助我们深入理解流处理领域的核心概念，为后续学习和使用其他流处理框架打下坚实基础。

思考题：在物联网设备监控场景中，需要实时处理千万级设备的温度、压力数据，进行异常检测和预警。你认为Storm是否适合这个场景？如果适合，你会如何设计拓扑结构？如果认为不适合，你会选择什么替代方案？欢迎在评论区分享你的见解！