HBase Compaction调优实战：Minor/Major Compaction触发条件与资源控制
HBase作为构建在Hadoop之上的NoSQL数据库，采用LSM树(Log-Structured Merge Tree)架构。这种架构通过将随机写转换为顺序写来提高写入性能，但随之而来的是Compaction机制。Compaction是HBase核心的后台操作，用于合并StoreFile、清理过期数据以及删除标记，从而提升读性能。
在HBase中，Compaction主要分为两种类型：Minor Compaction和Major Compaction。Minor Compaction将多个小的StoreFile合并为一个较大的StoreFile，但不会删除已标记删除的数据，也不会对数据进行版本合并。Major Compaction则将StoreFile中所有数据进行合并，包括清理已标记删除的数据、合并多个版本的数据，以及将数据按Region重新分区。Compaction对HBase性能有着双面影响：一方面可以优化读性能，减少文件数量；另一方面，它会消耗大量I/O和CPU资源，可能影响集群的整体性能。因此，合理配置Compaction策略至关重要。
Minor Compaction的主要目的是合并小文件，减少StoreFile数量，从而提升查询效率。其触发条件主要由以下参数控制：hbase.hstore.compactionThreshold是每个Store中StoreFile的数量阈值，当StoreFile数量超过此值时，会触发Minor Compaction，默认值为3；hbase.hstore.compaction.min是每次Minor Compaction最少合并的文件数，默认为3；hbase.hstore.compaction.max是每次Minor Compaction最多合并的文件数，默认为10。合理配置Minor Compaction参数可以有效平衡读写性能：对于写入量大但读取量小的场景，可以适当提高hbase.hstore.compactionThreshold，减少Compaction频率；根据集群负载情况，调整hbase.hstore.compaction.min和max，避免单次Compaction消耗过多资源；对于HBase 2.0及以上版本，可启用Tiered Compaction，根据数据访问模式调整合并策略。
Major Compaction是对StoreFile进行全面合并的操作，能有效减少文件数量，清理过期数据，但消耗资源较大。其触发条件主要有：时间触发通过参数hbase.hregion.majorcompaction控制，默认为7天；可通过HBase Shell或Admin API手动触发；当Region分裂时，会触发Major Compaction；当单个StoreFile大小超过hbase.hstore.compaction.max.size参数值时，会触发Major Compaction。Major Compaction调优需谨慎，避免对集群造成过大压力：根据业务特点和集群负载，适当调整hbase.hregion.majorcompaction参数值；通过配置Cron表达式，在业务低峰期执行Major Compaction；对于HBase 2.0+，可启用Off-Peak Compaction；通过hbase.regionserver.thread.compaction.large参数控制Major Compaction的并发线程数。
HBase提供了多种资源控制机制：通过hbase.regionserver.throttle.compaction参数控制Compaction的I/O速率；使用hbase.regionserver.global.memstore.size限制内存使用；通过调整hbase.regionserver.thread.compaction.small和hbase.regionserver.thread.compaction.large控制Compaction线程数。性能优化策略包括：在业务低峰期执行Compaction操作；使用专门的Region Server处理Compaction任务；将StoreFile放在SSD上，减少I/O等待时间；调整hbase.wal.sync和hbase.wal.asyncfsync参数，平衡数据安全性和性能。监控以下关键指标评估Compaction效果：Compaction队列长度、Compaction完成时间、StoreFile数量、磁盘I/O使用率。
某电商公司HBase集群因Compaction导致业务高峰期响应延迟。通过以下调优措施，问题得到解决：将Minor Compaction阈值从3调整为5，减少Compaction频率；将Major Compaction时间窗口调整为凌晨2-4点；增加Compaction专用Region Server；使用SSD存储StoreFile。调整后，查询延迟降低了40%，集群稳定性显著提升。以下为最小配置示例：<property><name>hbase.hstore.compactionThreshold</name><value>5</value></property><property><name>hbase.regionserver.thread.compaction.small</name><value>3</value></property><property><name>hbase.regionserver.thread.compaction.large</name><value>2</value></property><property><name>hbase.hregion.majorcompaction</name><value>2592000000</value></property><property><name>hbase.hregion.majorcompaction.interval</name><value>2592000000</value></property>注意事项包括：调整Compaction参数前，充分测试对业务的影响；在业务低峰期进行Major Compaction；监控Compaction资源使用情况，避免影响业务；根据业务特点选择合适的Compaction策略；定期检查StoreFile数量，避免文件过多导致查询性能下降。
#publish-mermaid-1788834556718-0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#publish-mermaid-1788834556718-0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#publish-mermaid-1788834556718-0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#publish-mermaid-1788834556718-0 .error-icon{fill:#552222;}#publish-mermaid-1788834556718-0 .error-text{fill:#552222;stroke:#552222;}#publish-mermaid-1788834556718-0 .edge-thickness-normal{stroke-width:1px;}#publish-mermaid-1788834556718-0 .edge-thickness-thick{stroke-width:3.5px;}#publish-mermaid-1788834556718-0 .edge-pattern-solid{stroke-dasharray:0;}#publish-mermaid-1788834556718-0 .edge-thickness-invisible{stroke-width:0;fill:none;}#publish-mermaid-1788834556718-0 .edge-pattern-dashed{stroke-dasharray:3;}#publish-mermaid-1788834556718-0 .edge-pattern-dotted{stroke-dasharray:2;}#publish-mermaid-1788834556718-0 .marker{fill:#333333;stroke:#333333;}#publish-mermaid-1788834556718-0 .marker.cross{stroke:#333333;}#publish-mermaid-1788834556718-0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#publish-mermaid-1788834556718-0 p{margin:0;}#publish-mermaid-1788834556718-0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#publish-mermaid-1788834556718-0 .cluster-label text{fill:#333;}#publish-mermaid-1788834556718-0 .cluster-label span{color:#333;}#publish-mermaid-1788834556718-0 .cluster-label span p{background-color:transparent;}#publish-mermaid-1788834556718-0 .label text,#publish-mermaid-1788834556718-0 span{fill:#333;color:#333;}#publish-mermaid-1788834556718-0 .node rect,#publish-mermaid-1788834556718-0 .node circle,#publish-mermaid-1788834556718-0 .node ellipse,#publish-mermaid-1788834556718-0 .node polygon,#publish-mermaid-1788834556718-0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788834556718-0 .rough-node .label text,#publish-mermaid-1788834556718-0 .node .label text,#publish-mermaid-1788834556718-0 .image-shape .label,#publish-mermaid-1788834556718-0 .icon-shape .label{text-anchor:middle;}#publish-mermaid-1788834556718-0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#publish-mermaid-1788834556718-0 .rough-node .label,#publish-mermaid-1788834556718-0 .node .label,#publish-mermaid-1788834556718-0 .image-shape .label,#publish-mermaid-1788834556718-0 .icon-shape .label{text-align:center;}#publish-mermaid-1788834556718-0 .node.clickable{cursor:pointer;}#publish-mermaid-1788834556718-0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#publish-mermaid-1788834556718-0 .arrowheadPath{fill:#333333;}#publish-mermaid-1788834556718-0 .edgePath .path{stroke:#333333;stroke-width:1px;}#publish-mermaid-1788834556718-0 .flowchart-link{stroke:#333333;fill:none;}#publish-mermaid-1788834556718-0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788834556718-0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#publish-mermaid-1788834556718-0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788834556718-0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#publish-mermaid-1788834556718-0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#publish-mermaid-1788834556718-0 .cluster text{fill:#333;}#publish-mermaid-1788834556718-0 .cluster span{color:#333;}#publish-mermaid-1788834556718-0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#publish-mermaid-1788834556718-0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#publish-mermaid-1788834556718-0 rect.text{fill:none;stroke-width:0;}#publish-mermaid-1788834556718-0 .icon-shape,#publish-mermaid-1788834556718-0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788834556718-0 .icon-shape p,#publish-mermaid-1788834556718-0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#publish-mermaid-1788834556718-0 .icon-shape .label rect,#publish-mermaid-1788834556718-0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788834556718-0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#publish-mermaid-1788834556718-0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#publish-mermaid-1788834556718-0 .node .neo-node{stroke:#9370DB;}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node rect,#publish-mermaid-1788834556718-0 [data-look=\”neo\”].cluster rect,#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node polygon{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].swimlane.cluster rect{filter:none;}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node path{stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node .outer-path{filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node .neo-line path{stroke:#9370DB;filter:none;}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node circle{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].node circle .state-start{fill:#000000;}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].icon-shape .icon{fill:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788834556718-0 [data-look=\”neo\”].icon-shape .icon-neo path{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788834556718-0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}超过阈值未超过阈值触发条件满足触发条件不满足
数据写入HBase
检查StoreFile数量
触发Minor Compaction
继续写入
合并小文件
检查是否需要Major Compaction
执行Major Compaction
合并所有StoreFile
清理过期数据
