Apache Flume 是一个分布式、可靠、可用的系统，用于高效地收集、聚合和移动大量日志数据。Flume 采用基于流的架构，由 Agent、Source、Channel 和 Sink 四个核心组件构成。Flume 的特点是配置简单、性能高，特别适合日志数据的实时采集。
Apache NiFi 则是一个功能强大的数据流处理系统，提供图形化界面来设计、控制和监控数据流。NiFi 基于"数据流优先"的理念，内置了数据路由、转换、路由和验证等多种处理器，具有丰富的数据操作能力。NiFi 的突出优势在于其可视化操作界面、细粒度的数据流控制和强大的数据追踪能力。
两种工具在核心设计理念上有明显差异：Flume 更专注于数据的高效传输，而 NiFi 则强调数据的灵活处理和管理。在容错机制方面，Flume 依赖 Channel 的持久化能力，而 NiFi 则通过"数据流历史"和"回滚"机制提供更强的数据可靠性保证。
从架构角度看，Flume 采用的是轻量级架构，组件间耦合度低，部署和扩展相对简单。Flume 的 Source-Channel-Sink 模型适合单向数据流场景，但缺乏复杂的数据处理能力。NiFi 则采用基于进程的架构，内置丰富的处理器组件，能够实现复杂的数据流处理逻辑，但资源消耗相对较大。
在性能方面，Flume 对高吞吐量场景表现出色，特别是在日志数据采集方面，其传输延迟低，资源占用少。而 NiFi 在处理复杂的数据转换和路由逻辑时更具优势，特别是在需要细粒度控制的数据处理场景中。
在易用性方面，NiFi 的图形化界面大大降低了使用门槛，非技术人员也能轻松设计和监控数据流。而 Flume 需要编写配置文件，对用户的技术要求较高。
适用场景上，Flume 适合大规模日志采集、实时数据传输等简单数据流场景；而 NiFi 更适合需要复杂数据处理、数据质量检查、细粒度控制的企业级数据流处理场景。
在实际应用中，Flume 和 NiFi 可以结合使用，构建混合架构以发挥各自优势。典型的混合架构设计包括：使用 Flume 进行数据采集和初步过滤，然后通过可靠的数据传输机制将数据传递给 NiFi，由 NiFi 进行复杂的数据处理、路由和分发。
在设计混合架构时，需要注意以下几点：
混合架构特别适合以下场景：需要处理多样化的数据源、需要复杂的数据转换逻辑、需要细粒度的数据控制和高可靠性的数据流处理。
以下是一个简单的混合架构实现示例，展示如何使用 Flume 采集数据并通过 NiFi 处理：
首先，配置一个 Flume Agent，用于从 Web 服务器采集访问日志：
# agent.sources = r1
# agent.channels = c1
# agent.sinks = k1
# Describe/configure the source
# agent.sources.r1.type = exec
# agent.sources.r1.command = tail -F /var/log/nginx/access.log
# Describe the channel
# agent.channels.c1.type = memory
# agent.channels.c1.capacity = 1000
# Describe the sink
# agent.sinks.k1.type = avro
# agent.sinks.k1.hostname = localhost
# agent.sinks.k1.port = 4545
# Bind the source and sink to the channel
# agent.sources.r1.channels = c1
# agent.sinks.k1.channel = c1
然后在 NiFi 中配置相应的处理器接收数据并处理：
{
"processors": [
{
"id": "01",
"type": "GetFile",
"properties": {
"input": "/tmp/input",
"keepEmptyFiles": false
}
},
{
"id": "02",
"type": "Attribute",
"properties": {
"attributes": {
"timestamp": "${now():format('yyyy-MM-dd HH:mm:ss')}"
}
}
},
{
"id": "03",
"type": "PutHDFS",
"properties": {
"hdfsPath": "/data/processed",
"fileName": "access_${timestamp}.log"
}
}
]
}
注意事项：
通过合理配置和监控，这种混合架构能够高效可靠地处理各类数据流场景。
#publish-mermaid-1788229187582-0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#publish-mermaid-1788229187582-0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#publish-mermaid-1788229187582-0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#publish-mermaid-1788229187582-0 .error-icon{fill:#552222;}#publish-mermaid-1788229187582-0 .error-text{fill:#552222;stroke:#552222;}#publish-mermaid-1788229187582-0 .edge-thickness-normal{stroke-width:1px;}#publish-mermaid-1788229187582-0 .edge-thickness-thick{stroke-width:3.5px;}#publish-mermaid-1788229187582-0 .edge-pattern-solid{stroke-dasharray:0;}#publish-mermaid-1788229187582-0 .edge-thickness-invisible{stroke-width:0;fill:none;}#publish-mermaid-1788229187582-0 .edge-pattern-dashed{stroke-dasharray:3;}#publish-mermaid-1788229187582-0 .edge-pattern-dotted{stroke-dasharray:2;}#publish-mermaid-1788229187582-0 .marker{fill:#333333;stroke:#333333;}#publish-mermaid-1788229187582-0 .marker.cross{stroke:#333333;}#publish-mermaid-1788229187582-0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#publish-mermaid-1788229187582-0 p{margin:0;}#publish-mermaid-1788229187582-0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#publish-mermaid-1788229187582-0 .cluster-label text{fill:#333;}#publish-mermaid-1788229187582-0 .cluster-label span{color:#333;}#publish-mermaid-1788229187582-0 .cluster-label span p{background-color:transparent;}#publish-mermaid-1788229187582-0 .label text,#publish-mermaid-1788229187582-0 span{fill:#333;color:#333;}#publish-mermaid-1788229187582-0 .node rect,#publish-mermaid-1788229187582-0 .node circle,#publish-mermaid-1788229187582-0 .node ellipse,#publish-mermaid-1788229187582-0 .node polygon,#publish-mermaid-1788229187582-0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788229187582-0 .rough-node .label text,#publish-mermaid-1788229187582-0 .node .label text,#publish-mermaid-1788229187582-0 .image-shape .label,#publish-mermaid-1788229187582-0 .icon-shape .label{text-anchor:middle;}#publish-mermaid-1788229187582-0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#publish-mermaid-1788229187582-0 .rough-node .label,#publish-mermaid-1788229187582-0 .node .label,#publish-mermaid-1788229187582-0 .image-shape .label,#publish-mermaid-1788229187582-0 .icon-shape .label{text-align:center;}#publish-mermaid-1788229187582-0 .node.clickable{cursor:pointer;}#publish-mermaid-1788229187582-0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#publish-mermaid-1788229187582-0 .arrowheadPath{fill:#333333;}#publish-mermaid-1788229187582-0 .edgePath .path{stroke:#333333;stroke-width:1px;}#publish-mermaid-1788229187582-0 .flowchart-link{stroke:#333333;fill:none;}#publish-mermaid-1788229187582-0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788229187582-0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#publish-mermaid-1788229187582-0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788229187582-0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#publish-mermaid-1788229187582-0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#publish-mermaid-1788229187582-0 .cluster text{fill:#333;}#publish-mermaid-1788229187582-0 .cluster span{color:#333;}#publish-mermaid-1788229187582-0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#publish-mermaid-1788229187582-0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#publish-mermaid-1788229187582-0 rect.text{fill:none;stroke-width:0;}#publish-mermaid-1788229187582-0 .icon-shape,#publish-mermaid-1788229187582-0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788229187582-0 .icon-shape p,#publish-mermaid-1788229187582-0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#publish-mermaid-1788229187582-0 .icon-shape .label rect,#publish-mermaid-1788229187582-0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788229187582-0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#publish-mermaid-1788229187582-0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#publish-mermaid-1788229187582-0 .node .neo-node{stroke:#9370DB;}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node rect,#publish-mermaid-1788229187582-0 [data-look=\”neo\”].cluster rect,#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node polygon{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].swimlane.cluster rect{filter:none;}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node path{stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node .outer-path{filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node .neo-line path{stroke:#9370DB;filter:none;}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node circle{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].node circle .state-start{fill:#000000;}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].icon-shape .icon{fill:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788229187582-0 [data-look=\”neo\”].icon-shape .icon-neo path{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788229187582-0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
数据源
Flume Agent采集数据
数据缓冲区
NiFi处理器
数据路由决策
数据存储
数据转换
数据分发
数据分析
数据可视化
