Hadoop输入格式深度解析：默认输入格式与自定义实战

Hadoop输入格式深度解析：默认输入格式与自定义实战

• • 引言：输入格式——MapReduce的数据入口
  • 一、Hadoop的默认输入格式：TextInputFormat
  • • 1.1 默认输入格式概述
    • 1.2 工作原理
    • 1.3 其他内置输入格式
    • 1.4 TextInputFormat的优缺点
  • 二、为什么需要自定义输入格式？
  • • 2.1 典型场景
    • 2.2 内置格式无法满足的需求
  • 三、自定义输入格式的核心接口
  • • 3.1 InputFormat类体系
    • 3.2 需要实现的核心方法
    • • **1. 继承FileInputFormat，重写createRecordReader**
      • **2. 实现RecordReader，定义解析逻辑**
  • 四、实战案例一：处理非标准行分隔符
  • • 4.1 场景描述
    • 4.2 自定义RecordReader实现
    • 4.3 自定义InputFormat
    • 4.4 在作业中使用
  • 五、实战案例二：处理固定宽度文件
  • • 5.1 场景描述
    • 5.2 自定义RecordReader实现
  • 六、实战案例三：合并小文件
  • • 6.1 场景描述
    • 6.2 自定义CombineFileInputFormat
    • 6.3 ByteArrayRecordReader实现
  • 七、自定义输入格式的最佳实践
  • • 7.1 性能考虑
    • 7.2 可配置性设计
    • 7.3 测试与验证
  • 八、总结
  • • 8.1 核心要点回顾
    • 8.2 选择指南

引言：输入格式——MapReduce的数据入口

在Hadoop MapReduce框架中，输入格式（InputFormat） 是数据处理的起点，它决定了如何读取、分割和解析输入数据。正确选择或设计输入格式，直接影响作业的并行度、数据本地性和处理效率。

本文将深入剖析Hadoop的默认输入格式TextInputFormat的工作原理，并通过完整的实战案例展示如何自定义输入格式来解决特定业务场景中的问题。

#mermaid-svg-yfRrJFnQQ4nqBZTr{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yfRrJFnQQ4nqBZTr .error-icon{fill:#552222;}#mermaid-svg-yfRrJFnQQ4nqBZTr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yfRrJFnQQ4nqBZTr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-yfRrJFnQQ4nqBZTr .marker.cross{stroke:#333333;}#mermaid-svg-yfRrJFnQQ4nqBZTr svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-yfRrJFnQQ4nqBZTr p{margin:0;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge{stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 text{fill:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth–1{stroke-width:17;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-0{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-0{stroke-width:14;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-1{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-1{stroke-width:11;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 text{fill:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-2{stroke-width:8;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-3{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-3{stroke-width:5;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-4{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-4{stroke-width:2;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-5{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-5{stroke-width:-1;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-6{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-6{stroke-width:-4;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-7{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-7{stroke-width:-7;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-8{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-8{stroke-width:-10;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-9{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-9{stroke-width:-13;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 polygon,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 text{fill:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .node-icon-10{font-size:40px;color:black;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge-depth-10{stroke-width:-16;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:lightgray;}#mermaid-svg-yfRrJFnQQ4nqBZTr .disabled text{fill:#efefef;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-root rect,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-root path,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-root circle,#mermaid-svg-yfRrJFnQQ4nqBZTr .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-root text{fill:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-root span{color:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .section-2 span{color:#ffffff;}#mermaid-svg-yfRrJFnQQ4nqBZTr .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-yfRrJFnQQ4nqBZTr .edge{fill:none;}#mermaid-svg-yfRrJFnQQ4nqBZTr .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-yfRrJFnQQ4nqBZTr :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

输入格式InputFormat

核心功能

验证输入路径

将文件切分为分片

提供RecordReader

记录解析逻辑

默认实现

TextInputFormat

KeyValueTextInputFormat

NLineInputFormat

SequenceFileInputFormat

自定义扩展

继承FileInputFormat

实现RecordReader

定义键值类型

控制分片逻辑

一、Hadoop的默认输入格式：TextInputFormat

1.1 默认输入格式概述

Hadoop的默认输入格式是TextInputFormat 。它是FileInputFormat的子类，专门用于处理纯文本文件。

核心特性：

• 将输入文件按行分割
• 每行作为一条记录
• 键（Key）：该行在文件中的字节偏移量（LongWritable类型）
• 值（Value）：该行的文本内容（Text类型）

// TextInputFormat的典型输出
// <0, "Hello Hadoop">
// <15, "Welcome to MapReduce">
// <35, "Data processing framework">

1.2 工作原理

#mermaid-svg-QCojR4EGc85f0Z9d{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-QCojR4EGc85f0Z9d .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-QCojR4EGc85f0Z9d .error-icon{fill:#552222;}#mermaid-svg-QCojR4EGc85f0Z9d .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-QCojR4EGc85f0Z9d .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-QCojR4EGc85f0Z9d .marker{fill:#333333;stroke:#333333;}#mermaid-svg-QCojR4EGc85f0Z9d .marker.cross{stroke:#333333;}#mermaid-svg-QCojR4EGc85f0Z9d svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-QCojR4EGc85f0Z9d p{margin:0;}#mermaid-svg-QCojR4EGc85f0Z9d .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-QCojR4EGc85f0Z9d .cluster-label text{fill:#333;}#mermaid-svg-QCojR4EGc85f0Z9d .cluster-label span{color:#333;}#mermaid-svg-QCojR4EGc85f0Z9d .cluster-label span p{background-color:transparent;}#mermaid-svg-QCojR4EGc85f0Z9d .label text,#mermaid-svg-QCojR4EGc85f0Z9d span{fill:#333;color:#333;}#mermaid-svg-QCojR4EGc85f0Z9d .node rect,#mermaid-svg-QCojR4EGc85f0Z9d .node circle,#mermaid-svg-QCojR4EGc85f0Z9d .node ellipse,#mermaid-svg-QCojR4EGc85f0Z9d .node polygon,#mermaid-svg-QCojR4EGc85f0Z9d .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-QCojR4EGc85f0Z9d .rough-node .label text,#mermaid-svg-QCojR4EGc85f0Z9d .node .label text,#mermaid-svg-QCojR4EGc85f0Z9d .image-shape .label,#mermaid-svg-QCojR4EGc85f0Z9d .icon-shape .label{text-anchor:middle;}#mermaid-svg-QCojR4EGc85f0Z9d .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-QCojR4EGc85f0Z9d .rough-node .label,#mermaid-svg-QCojR4EGc85f0Z9d .node .label,#mermaid-svg-QCojR4EGc85f0Z9d .image-shape .label,#mermaid-svg-QCojR4EGc85f0Z9d .icon-shape .label{text-align:center;}#mermaid-svg-QCojR4EGc85f0Z9d .node.clickable{cursor:pointer;}#mermaid-svg-QCojR4EGc85f0Z9d .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-QCojR4EGc85f0Z9d .arrowheadPath{fill:#333333;}#mermaid-svg-QCojR4EGc85f0Z9d .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-QCojR4EGc85f0Z9d .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-QCojR4EGc85f0Z9d .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QCojR4EGc85f0Z9d .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-QCojR4EGc85f0Z9d .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QCojR4EGc85f0Z9d .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-QCojR4EGc85f0Z9d .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-QCojR4EGc85f0Z9d .cluster text{fill:#333;}#mermaid-svg-QCojR4EGc85f0Z9d .cluster span{color:#333;}#mermaid-svg-QCojR4EGc85f0Z9d div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-QCojR4EGc85f0Z9d .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-QCojR4EGc85f0Z9d rect.text{fill:none;stroke-width:0;}#mermaid-svg-QCojR4EGc85f0Z9d .icon-shape,#mermaid-svg-QCojR4EGc85f0Z9d .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QCojR4EGc85f0Z9d .icon-shape p,#mermaid-svg-QCojR4EGc85f0Z9d .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-QCojR4EGc85f0Z9d .icon-shape rect,#mermaid-svg-QCojR4EGc85f0Z9d .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QCojR4EGc85f0Z9d .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-QCojR4EGc85f0Z9d .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-QCojR4EGc85f0Z9d :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

记录解析

TextInputFormat处理

输入文件

文件 data.txt512MB

分片计算默认等于块大小128MB

分片10-128MB

分片2128-256MB

分片3256-384MB

分片4384-512MB

LineRecordReader

LineRecordReader

LineRecordReader

LineRecordReader

逐行读取

<0,'line1'>

<15,'line2'>

关键组件：

1.3 其他内置输入格式

1.4 TextInputFormat的优缺点

优点：

• 简单通用，人类可读
• 支持所有文本格式
• 易于调试（可直接用hadoop fs -cat查看）

缺点：

• 存储效率低（比二进制格式多占用30-50%空间）
• 解析开销大（字符串转换、类型判断）
• 不保留数据类型信息
• 不支持复杂数据结构嵌套

二、为什么需要自定义输入格式？

2.1 典型场景

#mermaid-svg-JFGJOJZ4BoGwcFZf{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-JFGJOJZ4BoGwcFZf .error-icon{fill:#552222;}#mermaid-svg-JFGJOJZ4BoGwcFZf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-JFGJOJZ4BoGwcFZf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-JFGJOJZ4BoGwcFZf .marker.cross{stroke:#333333;}#mermaid-svg-JFGJOJZ4BoGwcFZf svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-JFGJOJZ4BoGwcFZf p{margin:0;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge{stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 text{fill:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth–1{stroke-width:17;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-0{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-0{stroke-width:14;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-1{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-1{stroke-width:11;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 text{fill:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-2{stroke-width:8;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-3{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-3{stroke-width:5;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-4{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-4{stroke-width:2;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-5{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-5{stroke-width:-1;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-6{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-6{stroke-width:-4;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-7{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-7{stroke-width:-7;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-8{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-8{stroke-width:-10;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-9{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-9{stroke-width:-13;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 polygon,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 text{fill:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .node-icon-10{font-size:40px;color:black;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge-depth-10{stroke-width:-16;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:lightgray;}#mermaid-svg-JFGJOJZ4BoGwcFZf .disabled text{fill:#efefef;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-root rect,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-root path,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-root circle,#mermaid-svg-JFGJOJZ4BoGwcFZf .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-root text{fill:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-root span{color:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .section-2 span{color:#ffffff;}#mermaid-svg-JFGJOJZ4BoGwcFZf .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-JFGJOJZ4BoGwcFZf .edge{fill:none;}#mermaid-svg-JFGJOJZ4BoGwcFZf .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-JFGJOJZ4BoGwcFZf :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

自定义输入格式场景

特殊文件格式

非标准分隔符

固定宽度记录

二进制格式

自定义序列化

特殊解析逻辑

跳过文件头

按业务规则分组

数据过滤

格式转换

性能优化

合并小文件

预聚合

索引读取

列式裁剪

2.2 内置格式无法满足的需求

三、自定义输入格式的核心接口

3.1 InputFormat类体系

#mermaid-svg-1vmsAe2cEnx8KmbH{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-1vmsAe2cEnx8KmbH .error-icon{fill:#552222;}#mermaid-svg-1vmsAe2cEnx8KmbH .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-1vmsAe2cEnx8KmbH .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-1vmsAe2cEnx8KmbH .marker{fill:#333333;stroke:#333333;}#mermaid-svg-1vmsAe2cEnx8KmbH .marker.cross{stroke:#333333;}#mermaid-svg-1vmsAe2cEnx8KmbH svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-1vmsAe2cEnx8KmbH p{margin:0;}#mermaid-svg-1vmsAe2cEnx8KmbH .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH .cluster-label text{fill:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH .cluster-label span{color:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH .cluster-label span p{background-color:transparent;}#mermaid-svg-1vmsAe2cEnx8KmbH .label text,#mermaid-svg-1vmsAe2cEnx8KmbH span{fill:#333;color:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH .node rect,#mermaid-svg-1vmsAe2cEnx8KmbH .node circle,#mermaid-svg-1vmsAe2cEnx8KmbH .node ellipse,#mermaid-svg-1vmsAe2cEnx8KmbH .node polygon,#mermaid-svg-1vmsAe2cEnx8KmbH .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-1vmsAe2cEnx8KmbH .rough-node .label text,#mermaid-svg-1vmsAe2cEnx8KmbH .node .label text,#mermaid-svg-1vmsAe2cEnx8KmbH .image-shape .label,#mermaid-svg-1vmsAe2cEnx8KmbH .icon-shape .label{text-anchor:middle;}#mermaid-svg-1vmsAe2cEnx8KmbH .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-1vmsAe2cEnx8KmbH .rough-node .label,#mermaid-svg-1vmsAe2cEnx8KmbH .node .label,#mermaid-svg-1vmsAe2cEnx8KmbH .image-shape .label,#mermaid-svg-1vmsAe2cEnx8KmbH .icon-shape .label{text-align:center;}#mermaid-svg-1vmsAe2cEnx8KmbH .node.clickable{cursor:pointer;}#mermaid-svg-1vmsAe2cEnx8KmbH .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-1vmsAe2cEnx8KmbH .arrowheadPath{fill:#333333;}#mermaid-svg-1vmsAe2cEnx8KmbH .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-1vmsAe2cEnx8KmbH .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-1vmsAe2cEnx8KmbH .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-1vmsAe2cEnx8KmbH .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-1vmsAe2cEnx8KmbH .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-1vmsAe2cEnx8KmbH .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-1vmsAe2cEnx8KmbH .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-1vmsAe2cEnx8KmbH .cluster text{fill:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH .cluster span{color:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-1vmsAe2cEnx8KmbH .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-1vmsAe2cEnx8KmbH rect.text{fill:none;stroke-width:0;}#mermaid-svg-1vmsAe2cEnx8KmbH .icon-shape,#mermaid-svg-1vmsAe2cEnx8KmbH .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-1vmsAe2cEnx8KmbH .icon-shape p,#mermaid-svg-1vmsAe2cEnx8KmbH .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-1vmsAe2cEnx8KmbH .icon-shape rect,#mermaid-svg-1vmsAe2cEnx8KmbH .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-1vmsAe2cEnx8KmbH .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-1vmsAe2cEnx8KmbH .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-1vmsAe2cEnx8KmbH :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

InputFormat抽象类

FileInputFormat基于文件的实现

DBInputFormat数据库输入

TextInputFormat默认

KeyValueTextInputFormat键值对

SequenceFileInputFormat序列文件

自定义InputFormat继承实现

3.2 需要实现的核心方法

自定义输入格式通常需要继承FileInputFormat并实现两个核心组件：

1. 继承FileInputFormat，重写createRecordReader

public class CustomInputFormat extends FileInputFormat<KeyType, ValueType> {

@Override
public RecordReader<KeyType, ValueType> createRecordReader(
InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {

// 返回自定义的RecordReader实例
return new CustomRecordReader();
}

// 可选：重写isSplitable控制文件是否可切分
@Override
protected boolean isSplitable(JobContext context, Path filename) {
// 对于不可切分的格式（如gzip压缩），返回false
return super.isSplitable(context, filename);
}

// 可选：重写getSplits控制分片逻辑
@Override
public List<InputSplit> getSplits(JobContext job) throws IOException {
// 自定义分片策略，如合并小文件
return super.getSplits(job);
}
}

2. 实现RecordReader，定义解析逻辑

public class CustomRecordReader extends RecordReader<KeyType, ValueType> {

// 初始化方法，在任务开始时调用
@Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
// 打开文件流，定位到分片起始位置
}

// 读取下一条记录，返回true表示还有记录
@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
// 读取下一条记录，填充key和value
}

// 获取当前键
@Override
public KeyType getCurrentKey() throws IOException, InterruptedException {
return key;
}

// 获取当前值
@Override
public ValueType getCurrentValue() throws IOException, InterruptedException {
return value;
}

// 获取处理进度
@Override
public float getProgress() throws IOException, InterruptedException {
// 返回0.0-1.0之间的进度值
}

// 关闭资源
@Override
public void close() throws IOException {
// 关闭文件流等资源
}
}

四、实战案例一：处理非标准行分隔符

4.1 场景描述

某些数据文件不是以换行符\\n分隔行，而是使用特殊字符组合如!@!作为行分隔符。默认的TextInputFormat无法正确解析这种文件。

4.2 自定义RecordReader实现

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;

import java.io.IOException;

public class CustomDelimiterRecordReader extends RecordReader<LongWritable, Text> {

private LongWritable key;
private Text value;
private FSDataInputStream inputStream;
private long start;
private long end;
private long pos;
private byte[] delimiter;
private int delimiterLength;

@Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {

// 获取分片信息
FileSplit fileSplit = (FileSplit) split;
Path path = fileSplit.getPath();
Configuration conf = context.getConfiguration();

// 获取自定义分隔符（可从配置中读取）
String delimStr = conf.get("custom.record.delimiter", "!@!");
delimiter = delimStr.getBytes("UTF-8");
delimiterLength = delimiter.length;

// 打开文件流
FileSystem fs = path.getFileSystem(conf);
inputStream = fs.open(path);

// 定位到分片起始位置
start = fileSplit.getStart();
end = start + fileSplit.getLength();
inputStream.seek(start);
pos = start;

key = new LongWritable();
value = new Text();
}

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (pos >= end) {
return false;
}

// 读取数据直到遇到分隔符或到达分片末尾
ByteArrayOutputStream buffer = new ByteArrayOutputStream();

// 处理可能的分片边界情况
boolean boundaryFound = false;
while (pos < end && !boundaryFound) {
int b = inputStream.read();
if (b == –1) {
break;
}
pos++;

buffer.write(b);

// 检查是否匹配分隔符
if (b == delimiter[delimiterLength – 1]) {
byte[] data = buffer.toByteArray();
if (endsWithDelimiter(data)) {
// 找到完整记录，去掉末尾的分隔符
byte[] record = new byte[data.length – delimiterLength];
System.arraycopy(data, 0, record, 0, data.length – delimiterLength);
value.set(record);
boundaryFound = true;
}
}
}

// 设置键为记录的起始偏移量
key.set(pos – buffer.size());

return true;
}

private boolean endsWithDelimiter(byte[] data) {
if (data.length < delimiterLength) {
return false;
}
for (int i = 0; i < delimiterLength; i++) {
if (data[data.length – delimiterLength + i] != delimiter[i]) {
return false;
}
}
return true;
}

@Override
public LongWritable getCurrentKey() {
return key;
}

@Override
public Text getCurrentValue() {
return value;
}

@Override
public float getProgress() {
if (start == end) {
return 0.0f;
}
return Math.min(1.0f, (pos – start) / (float) (end – start));
}

@Override
public void close() throws IOException {
if (inputStream != null) {
inputStream.close();
}
}
}

4.3 自定义InputFormat

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

public class CustomDelimiterInputFormat extends FileInputFormat<LongWritable, Text> {

@Override
public RecordReader<LongWritable, Text> createRecordReader(
InputSplit split, TaskAttemptContext context) {

// 返回自定义的RecordReader
return new CustomDelimiterRecordReader();
}

// 确保文件可切分
@Override
protected boolean isSplitable(JobContext context, Path filename) {
// 根据实际情况返回，如果是压缩文件可能返回false
return true;
}
}

4.4 在作业中使用

public class CustomDelimiterJob {

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();

// 设置自定义分隔符
conf.set("custom.record.delimiter", "!@!");

Job job = Job.getInstance(conf, "Custom Delimiter Job");

// 设置自定义输入格式
job.setInputFormatClass(CustomDelimiterInputFormat.class);

// 其他配置…
job.setJarByClass(CustomDelimiterJob.class);
job.setMapperClass(MyMapper.class);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

五、实战案例二：处理固定宽度文件

5.1 场景描述

传统系统导出的固定宽度文件，每条记录长度固定（如100字节），需要按列宽解析。

5.2 自定义RecordReader实现

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;

import java.io.IOException;

public class FixedWidthRecordReader extends RecordReader<LongWritable, Text> {

private LongWritable key;
private Text value;
private FSDataInputStream inputStream;
private long start;
private long end;
private long pos;
private int recordLength;
private byte[] recordBuffer;

@Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {

FileSplit fileSplit = (FileSplit) split;
Path path = fileSplit.getPath();
Configuration conf = context.getConfiguration();

// 获取记录长度配置（字节数）
recordLength = conf.getInt("fixed.record.length", 100);
recordBuffer = new byte[recordLength];

// 打开文件流
FileSystem fs = path.getFileSystem(conf);
inputStream = fs.open(path);

// 定位到分片起始位置
start = fileSplit.getStart();
end = start + fileSplit.getLength();

// 对齐到记录边界
if (start % recordLength != 0) {
// 如果不是记录边界，向前移动到下一条记录起始
start = start – (start % recordLength) + recordLength;
}

inputStream.seek(start);
pos = start;

key = new LongWritable();
value = new Text();
}

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (pos >= end) {
return false;
}

// 读取固定长度的记录
int bytesRead = inputStream.read(recordBuffer, 0, recordLength);
if (bytesRead < recordLength) {
return false; // 不完整的记录
}

// 设置键为记录起始偏移量
key.set(pos);

// 设置值为记录内容
value.set(recordBuffer, 0, recordLength);

pos += recordLength;

return true;
}

// 其他方法实现…
}

六、实战案例三：合并小文件

6.1 场景描述

HDFS中大量小文件会导致NameNode内存压力和过多的Map任务。CombineFileInputFormat可以将多个小文件合并到一个分片处理。

6.2 自定义CombineFileInputFormat

import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.CombineFileSplit;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;

import java.io.IOException;

public class CustomCombineInputFormat
extends CombineFileInputFormat<Text, BytesWritable> {

public CustomCombineInputFormat() {
// 设置最大分片大小（例如256MB）
super.setMaxSplitSize(268435456);
}

@Override
public RecordReader<Text, BytesWritable> createRecordReader(
InputSplit split, TaskAttemptContext context) throws IOException {

// 使用CombineFileRecordReader处理合并后的分片
return new CombineFileRecordReader<>(
(CombineFileSplit) split,
context,
ByteArrayRecordReader.class
);
}
}

6.3 ByteArrayRecordReader实现

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.input.CombineFileSplit;

import java.io.IOException;

public class ByteArrayRecordReader extends RecordReader<Text, BytesWritable> {

private CombineFileSplit split;
private int currentIndex;
private float currentProgress;
private Text currentKey;
private BytesWritable currentValue;

@Override
public void initialize(InputSplit genericSplit, TaskAttemptContext context)
throws IOException {
this.split = (CombineFileSplit) genericSplit;
this.currentIndex = 0;
this.currentProgress = 0;
}

@Override
public boolean nextKeyValue() throws IOException {
if (currentIndex >= split.getNumPaths()) {
return false;
}

// 获取当前文件的路径和偏移
Path filePath = split.getPath(currentIndex);
long offset = split.getOffset(currentIndex);
long length = split.getLength(currentIndex);

// 读取文件内容
Configuration conf = new Configuration();
FileSystem fs = filePath.getFileSystem(conf);
FSDataInputStream in = fs.open(filePath);

byte[] content = new byte[(int) length];
in.readFully(offset, content);
in.close();

// 设置键为文件名
currentKey = new Text(filePath.getName());

// 设置值为文件内容
currentValue = new BytesWritable(content);

currentIndex++;
currentProgress = (float) currentIndex / split.getNumPaths();

return true;
}

@Override
public Text getCurrentKey() {
return currentKey;
}

@Override
public BytesWritable getCurrentValue() {
return currentValue;
}

@Override
public float getProgress() {
return currentProgress;
}

@Override
public void close() {
// 不需要额外关闭
}
}

七、自定义输入格式的最佳实践

7.1 性能考虑

7.2 可配置性设计

public class ConfigurableRecordReader extends RecordReader {

private String encoding;
private String delimiter;
private boolean skipHeader;

@Override
public void initialize(InputSplit split, TaskAttemptContext context) {
Configuration conf = context.getConfiguration();

// 从配置读取参数，提供默认值
encoding = conf.get("custom.encoding", "UTF-8");
delimiter = conf.get("custom.delimiter", ",");
skipHeader = conf.getBoolean("custom.skip.header", false);

// 使用这些参数初始化
}
}

7.3 测试与验证

public class CustomInputFormatTest {

@Test
public void testRecordReader() throws Exception {
// 创建测试文件
Path testFile = new Path("test.txt");
FileSystem fs = FileSystem.getLocal(new Configuration());
FSDataOutputStream out = fs.create(testFile);
out.writeBytes("record1!@!record2!@!record3");
out.close();

// 创建分片
FileSplit split = new FileSplit(testFile, 0,
fs.getFileStatus(testFile).getLen(), new String[0]);

// 创建RecordReader
CustomDelimiterRecordReader reader = new CustomDelimiterRecordReader();
reader.initialize(split, new TaskAttemptContext());

// 验证读取结果
assertTrue(reader.nextKeyValue());
assertEquals(0L, reader.getCurrentKey().get());
assertEquals("record1", reader.getCurrentValue().toString());

// 清理
reader.close();
fs.delete(testFile, true);
}
}

八、总结

8.1 核心要点回顾

#mermaid-svg-bLkOsNPTeM3DkLCB{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-bLkOsNPTeM3DkLCB .error-icon{fill:#552222;}#mermaid-svg-bLkOsNPTeM3DkLCB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-bLkOsNPTeM3DkLCB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-bLkOsNPTeM3DkLCB .marker.cross{stroke:#333333;}#mermaid-svg-bLkOsNPTeM3DkLCB svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-bLkOsNPTeM3DkLCB p{margin:0;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge{stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 text{fill:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth–1{stroke-width:17;}#mermaid-svg-bLkOsNPTeM3DkLCB .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-0{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-0{stroke-width:14;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-1{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-1{stroke-width:11;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 text{fill:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-2{stroke-width:8;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-3{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-3{stroke-width:5;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-4{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-4{stroke-width:2;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-5{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-5{stroke-width:-1;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-6{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-6{stroke-width:-4;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-7{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-7{stroke-width:-7;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-8{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-8{stroke-width:-10;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-9{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-9{stroke-width:-13;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 polygon,#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 text{fill:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .node-icon-10{font-size:40px;color:black;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .edge-depth-10{stroke-width:-16;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled circle,#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:lightgray;}#mermaid-svg-bLkOsNPTeM3DkLCB .disabled text{fill:#efefef;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-root rect,#mermaid-svg-bLkOsNPTeM3DkLCB .section-root path,#mermaid-svg-bLkOsNPTeM3DkLCB .section-root circle,#mermaid-svg-bLkOsNPTeM3DkLCB .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-bLkOsNPTeM3DkLCB .section-root text{fill:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-root span{color:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .section-2 span{color:#ffffff;}#mermaid-svg-bLkOsNPTeM3DkLCB .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-bLkOsNPTeM3DkLCB .edge{fill:none;}#mermaid-svg-bLkOsNPTeM3DkLCB .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-bLkOsNPTeM3DkLCB :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

输入格式总结

默认格式

TextInputFormat

行偏移量为键

行内容为值

适用于文本文件

内置格式

KeyValueTextInputFormat

NLineInputFormat

SequenceFileInputFormat

CombineTextInputFormat

自定义场景

特殊分隔符

固定宽度

二进制格式

合并小文件

实现要点

继承FileInputFormat

实现RecordReader

处理分片边界

配置参数化

8.2 选择指南

核心启示：输入格式是MapReduce作业的"第一公里"。选择正确的输入格式能事半功倍，而自定义输入格式则是解决复杂数据格式问题的终极武器。理解其设计原理，能让你的Hadoop作业更加高效、健壮。

互动问题：你在实际项目中遇到过哪些需要自定义输入格式的场景？是处理XML/JSON多行记录，还是解析老旧系统的固定宽度文件？欢迎在评论区分享你的经验和解决方案！