25、Python批量处理1000份PDF，这样写代码才叫优雅

自动化办公｜Python批量处理1000份PDF，这样写代码才叫优雅（多线程/多进程 + 断点续传 + 自动校验）

关键词：Python批量处理PDF、多线程多进程、断点续传、PDF提取校验、自动化办公、进度条tqdm、日志输出 适合读者：Python自动化工程师、运维开发、数据分析师、行政/财务IT支持

目录

文章目录

• 自动化办公｜Python批量处理1000份PDF，这样写代码才叫优雅（多线程/多进程 + 断点续传 + 自动校验）
• • 目录
  • 1. 引言：从“手动打开1000个PDF”到“一键批量处理”
  • 2. 批量处理场景：财务发票 / 行政合同 / 运营报表
  • • 2.1 三类典型业务
    • 2.2 共性与挑战
  • 3. 多线程 vs 多进程：选对武器，速度翻倍
  • • 3.1 概念图解
    • 3.2 什么时候用多线程？什么时候用多进程？
    • 3.3 批量处理PDF属于哪一类？
    • 3.4 代码实现对比
    • • 多线程版（适用文本提取）
      • 多进程版（适用OCR或CPU重计算）
    • 3.5 实际加速测试（1000份数字生成PDF，每页提取文字）
  • 4. 断点续传：再也不用担心程序崩溃
  • • 4.1 问题场景
    • 4.2 设计思路
    • 4.3 代码实现（带JSON持久化）
  • 5. 结果校验：自动核对提取字段完整性
  • • 5.1 为什么需要校验？
    • 5.2 校验维度
    • 5.3 实现一个通用校验器
    • 5.4 集成到批量处理中
  • 6. 输出报告：错误统计、准确率分析
  • • 6.1 报告内容设计
    • 6.2 生成HTML报告（带图表）
  • 7. 完整代码：一体化批量处理脚本
  • • 7.1 如何使用
    • 7.2 脚本特性总结
  • 8. 总结与进阶建议
  • • 8.1 本文核心知识点回顾
    • 8.2 常见问题FAQ
    • 8.3 下一步优化方向

1. 引言：从“手动打开1000个PDF”到“一键批量处理”

想象一下这样的场景：财务部门每个月要处理上千张供应商发票PDF，需要提取发票号、金额、日期；法务需要从2000份合同中抽取出违约责任条款；运营需要汇总1000份周报中的关键KPI……如果还靠人工一份份打开、复制、粘贴，不仅效率低下，还极易出错。

Python批量处理PDF 是自动化办公的必备技能。但当你真正开始写脚本时，会遇到一系列问题：

• 处理1000个PDF，单线程要跑几个小时，CPU利用率却只有12%
• 运行到第600个文件时程序突然崩溃，前功尽弃
• 提取出的数据没有校验，不知道哪些文件可能漏了字段
• 没有日志，不知道哪几个文件失败了

本文将带你一步步构建一个生产级批量PDF处理流水线，涵盖：

• 多线程 vs 多进程 —— 何时用哪个，代码怎么写得干净
• 断点续传 —— 再也不用从头再来
• 自动校验与报告 —— 准确率、错误统计一目了然
• 完整代码 —— 带进度条、日志、异常隔离，可直接复用

💡 阅读完本文，你将能写出每小时处理2000+ PDF的稳定脚本，且具备完善的容错和监控能力。

2. 批量处理场景：财务发票 / 行政合同 / 运营报表

2.1 三类典型业务

#mermaid-svg-XxgsDJS4sTej2Qjl{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-XxgsDJS4sTej2Qjl .error-icon{fill:#552222;}#mermaid-svg-XxgsDJS4sTej2Qjl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-XxgsDJS4sTej2Qjl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-XxgsDJS4sTej2Qjl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-XxgsDJS4sTej2Qjl .marker.cross{stroke:#333333;}#mermaid-svg-XxgsDJS4sTej2Qjl svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-XxgsDJS4sTej2Qjl p{margin:0;}#mermaid-svg-XxgsDJS4sTej2Qjl .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl .cluster-label text{fill:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl .cluster-label span{color:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl .cluster-label span p{background-color:transparent;}#mermaid-svg-XxgsDJS4sTej2Qjl .label text,#mermaid-svg-XxgsDJS4sTej2Qjl span{fill:#333;color:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl .node rect,#mermaid-svg-XxgsDJS4sTej2Qjl .node circle,#mermaid-svg-XxgsDJS4sTej2Qjl .node ellipse,#mermaid-svg-XxgsDJS4sTej2Qjl .node polygon,#mermaid-svg-XxgsDJS4sTej2Qjl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-XxgsDJS4sTej2Qjl .rough-node .label text,#mermaid-svg-XxgsDJS4sTej2Qjl .node .label text,#mermaid-svg-XxgsDJS4sTej2Qjl .image-shape .label,#mermaid-svg-XxgsDJS4sTej2Qjl .icon-shape .label{text-anchor:middle;}#mermaid-svg-XxgsDJS4sTej2Qjl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-XxgsDJS4sTej2Qjl .rough-node .label,#mermaid-svg-XxgsDJS4sTej2Qjl .node .label,#mermaid-svg-XxgsDJS4sTej2Qjl .image-shape .label,#mermaid-svg-XxgsDJS4sTej2Qjl .icon-shape .label{text-align:center;}#mermaid-svg-XxgsDJS4sTej2Qjl .node.clickable{cursor:pointer;}#mermaid-svg-XxgsDJS4sTej2Qjl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-XxgsDJS4sTej2Qjl .arrowheadPath{fill:#333333;}#mermaid-svg-XxgsDJS4sTej2Qjl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-XxgsDJS4sTej2Qjl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-XxgsDJS4sTej2Qjl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-XxgsDJS4sTej2Qjl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-XxgsDJS4sTej2Qjl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-XxgsDJS4sTej2Qjl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-XxgsDJS4sTej2Qjl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-XxgsDJS4sTej2Qjl .cluster text{fill:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl .cluster span{color:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-XxgsDJS4sTej2Qjl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-XxgsDJS4sTej2Qjl rect.text{fill:none;stroke-width:0;}#mermaid-svg-XxgsDJS4sTej2Qjl .icon-shape,#mermaid-svg-XxgsDJS4sTej2Qjl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-XxgsDJS4sTej2Qjl .icon-shape p,#mermaid-svg-XxgsDJS4sTej2Qjl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-XxgsDJS4sTej2Qjl .icon-shape .label rect,#mermaid-svg-XxgsDJS4sTej2Qjl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-XxgsDJS4sTej2Qjl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-XxgsDJS4sTej2Qjl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-XxgsDJS4sTej2Qjl :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

运营报表

周报PDF

提取: 销售额/完成率/问题项

BI看板

行政合同

合同扫描件

提取: 签约方/有效期/违约金

合同管理系统

财务发票

供应商发票PDF

提取: 发票号/金额/税额/日期

自动录入ERP

2.2 共性与挑战

• 文件量大：少则几百，多则数万
• 格式相对统一：发票的发票号位置固定，合同的条款位置固定 → 可以模板化提取
• 单文件处理时间短（0.1~2秒），但总耗时线性增长
• 存在异常文件：损坏、加密、空白页、模板变更等

因此，批量处理的核心目标是：高吞吐 + 高鲁棒 + 可观测。

3. 多线程 vs 多进程：选对武器，速度翻倍

3.1 概念图解

#mermaid-svg-QHkh2QJFqaWAUwF0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-QHkh2QJFqaWAUwF0 .error-icon{fill:#552222;}#mermaid-svg-QHkh2QJFqaWAUwF0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-QHkh2QJFqaWAUwF0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .marker.cross{stroke:#333333;}#mermaid-svg-QHkh2QJFqaWAUwF0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-QHkh2QJFqaWAUwF0 p{margin:0;}#mermaid-svg-QHkh2QJFqaWAUwF0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .cluster-label text{fill:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .cluster-label span{color:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .cluster-label span p{background-color:transparent;}#mermaid-svg-QHkh2QJFqaWAUwF0 .label text,#mermaid-svg-QHkh2QJFqaWAUwF0 span{fill:#333;color:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .node rect,#mermaid-svg-QHkh2QJFqaWAUwF0 .node circle,#mermaid-svg-QHkh2QJFqaWAUwF0 .node ellipse,#mermaid-svg-QHkh2QJFqaWAUwF0 .node polygon,#mermaid-svg-QHkh2QJFqaWAUwF0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .rough-node .label text,#mermaid-svg-QHkh2QJFqaWAUwF0 .node .label text,#mermaid-svg-QHkh2QJFqaWAUwF0 .image-shape .label,#mermaid-svg-QHkh2QJFqaWAUwF0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-QHkh2QJFqaWAUwF0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .rough-node .label,#mermaid-svg-QHkh2QJFqaWAUwF0 .node .label,#mermaid-svg-QHkh2QJFqaWAUwF0 .image-shape .label,#mermaid-svg-QHkh2QJFqaWAUwF0 .icon-shape .label{text-align:center;}#mermaid-svg-QHkh2QJFqaWAUwF0 .node.clickable{cursor:pointer;}#mermaid-svg-QHkh2QJFqaWAUwF0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .arrowheadPath{fill:#333333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QHkh2QJFqaWAUwF0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-QHkh2QJFqaWAUwF0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QHkh2QJFqaWAUwF0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-QHkh2QJFqaWAUwF0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .cluster text{fill:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 .cluster span{color:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-QHkh2QJFqaWAUwF0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-QHkh2QJFqaWAUwF0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-QHkh2QJFqaWAUwF0 .icon-shape,#mermaid-svg-QHkh2QJFqaWAUwF0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QHkh2QJFqaWAUwF0 .icon-shape p,#mermaid-svg-QHkh2QJFqaWAUwF0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-QHkh2QJFqaWAUwF0 .icon-shape .label rect,#mermaid-svg-QHkh2QJFqaWAUwF0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QHkh2QJFqaWAUwF0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-QHkh2QJFqaWAUwF0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-QHkh2QJFqaWAUwF0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

多进程

进程1

独立内存

进程2

独立内存

进程3

独立内存

适合CPU密集型

多线程

线程1

共享内存/全局解释器锁GIL

线程2

线程3

交替执行, 适合I/O密集型

单线程

任务1

任务2

任务3

…

3.2 什么时候用多线程？什么时候用多进程？

3.3 批量处理PDF属于哪一类？

答案是：混合型，但整体偏I/O密集。

• 读取PDF文件（磁盘I/O）→ I/O密集
• 解析PDF文本（如pymupdf提取文字）→ 轻度CPU密集
• 如果需要OCR（扫描件）→ CPU+GPU密集，此时多进程优势明显

经验法则：

• 处理数字生成PDF（直接提取文本）：用多线程，线程数 = 2×CPU核心数 到 4×核心数
• 处理扫描版PDF（需OCR）：用多进程，进程数 = CPU核心数 或 GPU数量×2

3.4 代码实现对比

多线程版（适用文本提取）

from concurrent.futures import ThreadPoolExecutor, as_completed
import fitz # pymupdf

def extract_text_from_pdf(pdf_path):
with fitz.open(pdf_path) as doc:
text = ""
for page in doc:
text += page.get_text()
return text

def batch_multithread(pdf_list, max_workers=8):
results = {}
with ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_path = {executor.submit(extract_text_from_pdf, path): path
for path in pdf_list}
for future in as_completed(future_to_path):
path = future_to_path[future]
try:
results[path] = future.result()
except Exception as e:
results[path] = f"ERROR: {e}"
return results

多进程版（适用OCR或CPU重计算）

from concurrent.futures import ProcessPoolExecutor

def ocr_pdf(pdf_path):
# 假设使用PaddleOCR
from paddleocr import PaddleOCR
ocr = PaddleOCR(use_gpu=False) # 每个进程独立初始化
result = ocr.ocr(pdf_path)
return result

def batch_multiprocess(pdf_list, max_workers=4):
with ProcessPoolExecutor(max_workers=max_workers) as executor:
return list(executor.map(ocr_pdf, pdf_list))

⚠️ 注意：多进程下，OCR模型会在每个进程中重新加载，内存消耗会乘以进程数。如果使用GPU，需要确保每个进程绑定不同GPU或使用显存共享技术。

3.5 实际加速测试（1000份数字生成PDF，每页提取文字）

结论：对于纯文本提取，多线程优于多进程。

4. 断点续传：再也不用担心程序崩溃

4.1 问题场景

批量处理1000个文件，已经跑了800个，结果因为第801个文件损坏导致程序抛出未捕获异常直接退出。没有断点续传机制的话，修复后需要从头重新跑800个文件，浪费大量时间。

4.2 设计思路

用一个持久化的记录文件（如progress.json或sqlite）保存已经成功处理的文件列表。每次启动时先加载记录，跳过已处理的文件。

#mermaid-svg-a3GwJkZqxOTcrmwy{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-a3GwJkZqxOTcrmwy .error-icon{fill:#552222;}#mermaid-svg-a3GwJkZqxOTcrmwy .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-a3GwJkZqxOTcrmwy .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-a3GwJkZqxOTcrmwy .marker{fill:#333333;stroke:#333333;}#mermaid-svg-a3GwJkZqxOTcrmwy .marker.cross{stroke:#333333;}#mermaid-svg-a3GwJkZqxOTcrmwy svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-a3GwJkZqxOTcrmwy p{margin:0;}#mermaid-svg-a3GwJkZqxOTcrmwy .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy .cluster-label text{fill:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy .cluster-label span{color:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy .cluster-label span p{background-color:transparent;}#mermaid-svg-a3GwJkZqxOTcrmwy .label text,#mermaid-svg-a3GwJkZqxOTcrmwy span{fill:#333;color:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy .node rect,#mermaid-svg-a3GwJkZqxOTcrmwy .node circle,#mermaid-svg-a3GwJkZqxOTcrmwy .node ellipse,#mermaid-svg-a3GwJkZqxOTcrmwy .node polygon,#mermaid-svg-a3GwJkZqxOTcrmwy .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-a3GwJkZqxOTcrmwy .rough-node .label text,#mermaid-svg-a3GwJkZqxOTcrmwy .node .label text,#mermaid-svg-a3GwJkZqxOTcrmwy .image-shape .label,#mermaid-svg-a3GwJkZqxOTcrmwy .icon-shape .label{text-anchor:middle;}#mermaid-svg-a3GwJkZqxOTcrmwy .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-a3GwJkZqxOTcrmwy .rough-node .label,#mermaid-svg-a3GwJkZqxOTcrmwy .node .label,#mermaid-svg-a3GwJkZqxOTcrmwy .image-shape .label,#mermaid-svg-a3GwJkZqxOTcrmwy .icon-shape .label{text-align:center;}#mermaid-svg-a3GwJkZqxOTcrmwy .node.clickable{cursor:pointer;}#mermaid-svg-a3GwJkZqxOTcrmwy .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-a3GwJkZqxOTcrmwy .arrowheadPath{fill:#333333;}#mermaid-svg-a3GwJkZqxOTcrmwy .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-a3GwJkZqxOTcrmwy .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-a3GwJkZqxOTcrmwy .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-a3GwJkZqxOTcrmwy .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-a3GwJkZqxOTcrmwy .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-a3GwJkZqxOTcrmwy .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-a3GwJkZqxOTcrmwy .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-a3GwJkZqxOTcrmwy .cluster text{fill:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy .cluster span{color:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-a3GwJkZqxOTcrmwy .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-a3GwJkZqxOTcrmwy rect.text{fill:none;stroke-width:0;}#mermaid-svg-a3GwJkZqxOTcrmwy .icon-shape,#mermaid-svg-a3GwJkZqxOTcrmwy .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-a3GwJkZqxOTcrmwy .icon-shape p,#mermaid-svg-a3GwJkZqxOTcrmwy .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-a3GwJkZqxOTcrmwy .icon-shape .label rect,#mermaid-svg-a3GwJkZqxOTcrmwy .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-a3GwJkZqxOTcrmwy .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-a3GwJkZqxOTcrmwy .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-a3GwJkZqxOTcrmwy :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

存在

不存在

是

否

是

否

启动批量任务

检查progress文件

加载已处理文件集合

初始化为空集合

遍历所有待处理文件

文件已在已处理集合?

跳过

处理文件

处理成功?

将文件加入已处理集合实时保存progress

记录错误日志, 不加入集合

继续下一个

4.3 代码实现（带JSON持久化）

import json
import os
from pathlib import Path

class ResumeableBatchProcessor:
def __init__(self, progress_file="processing_progress.json"):
self.progress_file = progress_file
self.processed = self._load_progress()

def _load_progress(self):
if os.path.exists(self.progress_file):
with open(self.progress_file, 'r') as f:
return set(json.load(f))
return set()

def _save_progress(self):
with open(self.progress_file, 'w') as f:
json.dump(list(self.processed), f)

def mark_done(self, file_path):
self.processed.add(str(file_path))
self._save_progress() # 每条都保存，确保崩溃时损失最小

def is_done(self, file_path):
return str(file_path) in self.processed

def clear_progress(self):
"""重置进度（重新开始）"""
self.processed.clear()
self._save_progress()

# 使用示例
processor = ResumeableBatchProcessor()
pdf_files = list(Path("./pdfs").glob("*.pdf"))

for pdf in pdf_files:
if processor.is_done(pdf):
print(f"跳过已处理: {pdf.name}")
continue
try:
# 实际提取逻辑
result = extract_text_from_pdf(pdf)
# 保存结果…
processor.mark_done(pdf)
except Exception as e:
print(f"处理失败 {pdf.name}: {e}")
# 不标记已完成，下次重试

进阶：对于超大批量（10万+），建议使用SQLite代替JSON，避免每次都读写整个文件。

import sqlite3

class SQLiteResume:
def __init__(self, db_path="progress.db"):
self.conn = sqlite3.connect(db_path)
self.conn.execute("CREATE TABLE IF NOT EXISTS done (path TEXT PRIMARY KEY)")
self.conn.commit()

def is_done(self, path):
cur = self.conn.execute("SELECT 1 FROM done WHERE path=?", (str(path),))
return cur.fetchone() is not None

def mark_done(self, path):
self.conn.execute("INSERT OR IGNORE INTO done VALUES (?)", (str(path),))
self.conn.commit()

5. 结果校验：自动核对提取字段完整性

5.1 为什么需要校验？

提取出来的数据可能不完整：

• 发票PDF缺失“发票号”（可能是模板变化或扫描遮挡）
• 合同提取的“金额”字段是空字符串
• 日期格式不符合预期（2023-13-01）

人工逐条检查不现实，必须自动化校验。

5.2 校验维度

5.3 实现一个通用校验器

import re
from datetime import datetime

class FieldValidator:
@staticmethod
def not_empty(value, field_name):
if value is None or (isinstance(value, str) and value.strip() == ""):
raise ValueError(f"{field_name} 为空")
return True

@staticmethod
def regex(value, pattern, field_name):
if not re.match(pattern, str(value)):
raise ValueError(f"{field_name} 格式不匹配: {value}")
return True

@staticmethod
def is_date(value, fmt="%Y-%m-%d"):
try:
datetime.strptime(value, fmt)
return True
except:
raise ValueError(f"{field_name} 不是有效日期: {value}")

@staticmethod
def is_positive_number(value, field_name):
if not isinstance(value, (int, float)) or value < 0:
raise ValueError(f"{field_name} 应为正数: {value}")
return True

def validate_invoice(data):
"""校验发票提取结果"""
errors = []
try:
FieldValidator.not_empty(data.get("invoice_no"), "发票号")
FieldValidator.regex(data["invoice_no"], r'^\\d{10}$', "发票号")
FieldValidator.is_date(data.get("date"), "%Y-%m-%d")
FieldValidator.is_positive_number(data.get("amount"), "金额")
except ValueError as e:
errors.append(str(e))
return len(errors) == 0, errors

5.4 集成到批量处理中

在每个文件提取完成后立即校验，将校验结果存入结果字典。

results = []
for pdf in pdf_files:
raw_data = extract_invoice(pdf)
is_valid, errs = validate_invoice(raw_data)
results.append({
"file": str(pdf),
"status": "valid" if is_valid else "invalid",
"errors": errs,
"data": raw_data if is_valid else None
})

6. 输出报告：错误统计、准确率分析

处理完上千个文件后，你需要给业务方或领导一份清晰的报告：多少个成功？多少个失败？失败原因是什么？整体提取准确率如何？

6.1 报告内容设计

#mermaid-svg-J8HgSshFJnHbT0sE{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-J8HgSshFJnHbT0sE .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-J8HgSshFJnHbT0sE .error-icon{fill:#552222;}#mermaid-svg-J8HgSshFJnHbT0sE .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-J8HgSshFJnHbT0sE .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-J8HgSshFJnHbT0sE .marker{fill:#333333;stroke:#333333;}#mermaid-svg-J8HgSshFJnHbT0sE .marker.cross{stroke:#333333;}#mermaid-svg-J8HgSshFJnHbT0sE svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-J8HgSshFJnHbT0sE p{margin:0;}#mermaid-svg-J8HgSshFJnHbT0sE .edge{stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .section–1 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section–1 path,#mermaid-svg-J8HgSshFJnHbT0sE .section–1 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section–1 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section–1 text{fill:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth–1{stroke-width:17;}#mermaid-svg-J8HgSshFJnHbT0sE .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-0 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-0 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-0 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-0 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-0 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-0{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-0{stroke-width:14;}#mermaid-svg-J8HgSshFJnHbT0sE .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-1 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-1 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-1 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-1 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-1 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-1{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-1{stroke-width:11;}#mermaid-svg-J8HgSshFJnHbT0sE .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-2 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-2 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-2 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-2 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-2 text{fill:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-2{stroke-width:8;}#mermaid-svg-J8HgSshFJnHbT0sE .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-3 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-3 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-3 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-3 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-3 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-3{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-3{stroke-width:5;}#mermaid-svg-J8HgSshFJnHbT0sE .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-4 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-4 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-4 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-4 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-4 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-4{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-4{stroke-width:2;}#mermaid-svg-J8HgSshFJnHbT0sE .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-5 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-5 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-5 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-5 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-5 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-5{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-5{stroke-width:-1;}#mermaid-svg-J8HgSshFJnHbT0sE .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-6 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-6 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-6 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-6 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-6 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-6{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-6{stroke-width:-4;}#mermaid-svg-J8HgSshFJnHbT0sE .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-7 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-7 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-7 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-7 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-7 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-7{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-7{stroke-width:-7;}#mermaid-svg-J8HgSshFJnHbT0sE .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-8 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-8 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-8 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-8 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-8 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-8{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-8{stroke-width:-10;}#mermaid-svg-J8HgSshFJnHbT0sE .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-9 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-9 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-9 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-9 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-9 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-9{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-9{stroke-width:-13;}#mermaid-svg-J8HgSshFJnHbT0sE .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-10 rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-10 path,#mermaid-svg-J8HgSshFJnHbT0sE .section-10 circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-10 polygon,#mermaid-svg-J8HgSshFJnHbT0sE .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-10 text{fill:black;}#mermaid-svg-J8HgSshFJnHbT0sE .node-icon-10{font-size:40px;color:black;}#mermaid-svg-J8HgSshFJnHbT0sE .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .edge-depth-10{stroke-width:-16;}#mermaid-svg-J8HgSshFJnHbT0sE .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled,#mermaid-svg-J8HgSshFJnHbT0sE .disabled circle,#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:lightgray;}#mermaid-svg-J8HgSshFJnHbT0sE .disabled text{fill:#efefef;}#mermaid-svg-J8HgSshFJnHbT0sE .section-root rect,#mermaid-svg-J8HgSshFJnHbT0sE .section-root path,#mermaid-svg-J8HgSshFJnHbT0sE .section-root circle,#mermaid-svg-J8HgSshFJnHbT0sE .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-J8HgSshFJnHbT0sE .section-root text{fill:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .section-root span{color:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .section-2 span{color:#ffffff;}#mermaid-svg-J8HgSshFJnHbT0sE .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-J8HgSshFJnHbT0sE .edge{fill:none;}#mermaid-svg-J8HgSshFJnHbT0sE .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-J8HgSshFJnHbT0sE :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

批量处理报告

总体统计

总文件数

成功数/失败数

成功率

总耗时/平均每文件

错误分类

文件损坏

加密未解密

字段缺失Top3

格式错误Top3

准确率分析

关键字段完整率

字段格式正确率

详细清单

失败文件列表

成功但警告的文件

6.2 生成HTML报告（带图表）

使用pandas统计 + matplotlib生成简易图表，最后输出HTML。

import pandas as pd
import matplotlib.pyplot as plt
from datetime import datetime

def generate_report(results, output_dir="report"):
os.makedirs(output_dir, exist_ok=True)
df = pd.DataFrame(results)

# 基本统计
total = len(df)
success = df[df['status'] == 'valid'].shape[0]
failed = total – success
success_rate = success / total * 100

# 错误原因统计
error_counts = {}
for err_list in df[df['status'] == 'invalid']['errors']:
for err in err_list:
error_counts[err] = error_counts.get(err, 0) + 1
error_df = pd.DataFrame(error_counts.items(), columns=['错误类型', '次数']).sort_values('次数', ascending=False)

# 绘制饼图
plt.figure(figsize=(6,6))
plt.pie([success, failed], labels=['成功', '失败'], autopct='%1.1f%%', colors=['#4CAF50','#F44336'])
plt.title('处理成功率')
plt.savefig(f"{output_dir}/success_rate.png")

# 生成HTML
html = f"""
<html>
<head><meta charset="UTF-8"><title>PDF批量处理报告</title></head>
<body>
<h1>PDF批量处理报告</h1>
<p>生成时间：{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}</p>
<h2>总体概况</h2>
<ul>
<li>总文件数: {total}</li>
<li>成功处理: {success}</li>
<li>失败: {failed}</li>
<li>成功率: {success_rate:.2f}%</li>
</ul>
<img src="success_rate.png" />
<h2>错误分布</h2>
{error_df.to_html()}
<h2>失败文件列表</h2>
{df[df['status'] == 'invalid'][['file', 'errors']].to_html()}
</body>
</html>
"""
with open(f"{output_dir}/report.html", "w", encoding="utf-8") as f:
f.write(html)
print(f"报告已生成: {output_dir}/report.html")

7. 完整代码：一体化批量处理脚本

下面提供一个开箱即用的完整脚本，集成了：

• 多线程批量处理
• 断点续传（SQLite）
• 结果校验
• 进度条（tqdm）
• 日志输出（logging）
• 最终报告

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
批量PDF处理脚本 – 支持多线程、断点续传、自动校验、报告生成
"""

import os
import logging
import json
import sqlite3
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
from typing import Dict, List, Tuple

import fitz # pymupdf
from tqdm import tqdm

# ==================== 配置 ====================
PDF_DIR = "./pdfs"
OUTPUT_DIR = "./output"
MAX_WORKERS = 8
PROGRESS_DB = f"{OUTPUT_DIR}/progress.db"
LOG_FILE = f"{OUTPUT_DIR}/batch.log"

os.makedirs(OUTPUT_DIR, exist_ok=True)

# ==================== 日志配置 ====================
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s – %(levelname)s – %(message)s',
handlers=[
logging.FileHandler(LOG_FILE, encoding='utf-8'),
logging.StreamHandler()
]
)
logger = logging.getLogger(__name__)

# ==================== 断点续传（SQLite版） ====================
class ProgressTracker:
def __init__(self, db_path):
self.conn = sqlite3.connect(db_path)
self.conn.execute("CREATE TABLE IF NOT EXISTS done (file_path TEXT PRIMARY KEY, timestamp TEXT)")
self.conn.commit()

def is_done(self, file_path):
cur = self.conn.execute("SELECT 1 FROM done WHERE file_path=?", (str(file_path),))
return cur.fetchone() is not None

def mark_done(self, file_path):
self.conn.execute("INSERT OR REPLACE INTO done VALUES (?, ?)",
(str(file_path), datetime.now().isoformat()))
self.conn.commit()

def get_all_done(self):
cur = self.conn.execute("SELECT file_path FROM done")
return [row[0] for row in cur.fetchall()]

# ==================== PDF提取逻辑（示例：提取文本） ====================
def extract_content_from_pdf(pdf_path: Path) –> Dict:
"""
从PDF中提取需要的字段
实际使用时根据业务修改
"""
doc = fitz.open(pdf_path)
full_text = ""
for page in doc:
full_text += page.get_text()
doc.close()

# 模拟提取字段（示例）
# 实际可以用正则或模板提取发票号、金额等
result = {
"file_name": pdf_path.name,
"page_count": len(doc),
"text_length": len(full_text),
"full_text": full_text[:500] # 只保存前500字符作为预览
}
return result

# ==================== 字段校验 ====================
def validate_result(data: Dict) –> Tuple[bool, List[str]]:
errors = []
if data.get("page_count", 0) == 0:
errors.append("PDF页数为0")
if data.get("text_length", 0) < 10:
errors.append("提取文本过短（可能为扫描件）")
return len(errors) == 0, errors

# ==================== 处理单个文件（带重试） ====================
def process_one_file(pdf_path: Path, max_retries=2) –> Dict:
for attempt in range(max_retries):
try:
data = extract_content_from_pdf(pdf_path)
is_valid, errors = validate_result(data)
return {
"file": str(pdf_path),
"status": "valid" if is_valid else "invalid",
"errors": errors,
"data": data,
"attempt": attempt+1
}
except Exception as e:
logger.warning(f"处理 {pdf_path} 尝试 {attempt+1} 失败: {e}")
if attempt == max_retries – 1:
return {
"file": str(pdf_path),
"status": "error",
"errors": [str(e)],
"data": None,
"attempt": attempt+1
}
return None # unreachable

# ==================== 主批量处理函数 ====================
def batch_process():
logger.info("开始批量处理PDF")
# 获取所有PDF文件
pdf_files = list(Path(PDF_DIR).glob("*.pdf"))
logger.info(f"发现 {len(pdf_files)} 个PDF文件")

# 初始化进度追踪
tracker = ProgressTracker(PROGRESS_DB)
remaining_files = [f for f in pdf_files if not tracker.is_done(f)]
logger.info(f"跳过已处理 {len(pdf_files)–len(remaining_files)} 个，剩余 {len(remaining_files)} 个")

if not remaining_files:
logger.info("所有文件已处理完毕，无需运行")
return

results = []
# 使用多线程 + tqdm进度条
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
future_to_file = {executor.submit(process_one_file, f): f for f in remaining_files}
with tqdm(total=len(remaining_files), desc="处理PDF", unit="file") as pbar:
for future in as_completed(future_to_file):
result = future.result()
results.append(result)
if result["status"] == "valid":
tracker.mark_done(result["file"])
pbar.set_postfix({"成功": True})
else:
logger.error(f"处理失败: {result['file']}, 错误: {result['errors']}")
pbar.set_postfix({"成功": False})
pbar.update(1)

# 保存详细结果到JSON
results_json_path = f"{OUTPUT_DIR}/batch_results_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
with open(results_json_path, "w", encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False, indent=2)
logger.info(f"详细结果已保存至 {results_json_path}")

# 生成报告
generate_summary_report(results, OUTPUT_DIR)
logger.info("批量处理完成")

# ==================== 生成汇总报告 ====================
def generate_summary_report(results: List[Dict], output_dir: str):
total = len(results)
valid = sum(1 for r in results if r["status"] == "valid")
invalid = sum(1 for r in results if r["status"] == "invalid")
error = sum(1 for r in results if r["status"] == "error")

# 统计错误类型
error_type_count = {}
for r in results:
if r["status"] in ("invalid", "error"):
for err in r["errors"]:
error_type_count[err] = error_type_count.get(err, 0) + 1

# 生成Markdown报告
report_path = f"{output_dir}/SUMMARY.md"
with open(report_path, "w", encoding="utf-8") as f:
f.write(f"# PDF批量处理报告\\n\\n")
f.write(f"**生成时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\\n\\n")
f.write("## 总体统计\\n")
f.write(f"- 总文件数: {total}\\n")
f.write(f"- 成功(校验通过): {valid}\\n")
f.write(f"- 成功但校验不通过: {invalid}\\n")
f.write(f"- 彻底失败(异常): {error}\\n")
f.write(f"- 有效成功率: {valid/total*100:.2f}%\\n\\n")
f.write("## 错误分布\\n")
for err, cnt in sorted(error_type_count.items(), key=lambda x: –x[1]):
f.write(f"- {err}: {cnt}次\\n")
f.write("\\n## 失败文件列表\\n")
for r in results:
if r["status"] != "valid":
f.write(f"- {r['file']} : {', '.join(r['errors'])}\\n")
logger.info(f"报告已生成: {report_path}")

# ==================== 入口 ====================
if __name__ == "__main__":
batch_process()

7.1 如何使用

7.2 脚本特性总结

8. 总结与进阶建议

8.1 本文核心知识点回顾

#mermaid-svg-76LTDhLqxUxVTDcj{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-76LTDhLqxUxVTDcj .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-76LTDhLqxUxVTDcj .error-icon{fill:#552222;}#mermaid-svg-76LTDhLqxUxVTDcj .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-76LTDhLqxUxVTDcj .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-76LTDhLqxUxVTDcj .marker{fill:#333333;stroke:#333333;}#mermaid-svg-76LTDhLqxUxVTDcj .marker.cross{stroke:#333333;}#mermaid-svg-76LTDhLqxUxVTDcj svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-76LTDhLqxUxVTDcj p{margin:0;}#mermaid-svg-76LTDhLqxUxVTDcj .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-76LTDhLqxUxVTDcj .cluster-label text{fill:#333;}#mermaid-svg-76LTDhLqxUxVTDcj .cluster-label span{color:#333;}#mermaid-svg-76LTDhLqxUxVTDcj .cluster-label span p{background-color:transparent;}#mermaid-svg-76LTDhLqxUxVTDcj .label text,#mermaid-svg-76LTDhLqxUxVTDcj span{fill:#333;color:#333;}#mermaid-svg-76LTDhLqxUxVTDcj .node rect,#mermaid-svg-76LTDhLqxUxVTDcj .node circle,#mermaid-svg-76LTDhLqxUxVTDcj .node ellipse,#mermaid-svg-76LTDhLqxUxVTDcj .node polygon,#mermaid-svg-76LTDhLqxUxVTDcj .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-76LTDhLqxUxVTDcj .rough-node .label text,#mermaid-svg-76LTDhLqxUxVTDcj .node .label text,#mermaid-svg-76LTDhLqxUxVTDcj .image-shape .label,#mermaid-svg-76LTDhLqxUxVTDcj .icon-shape .label{text-anchor:middle;}#mermaid-svg-76LTDhLqxUxVTDcj .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-76LTDhLqxUxVTDcj .rough-node .label,#mermaid-svg-76LTDhLqxUxVTDcj .node .label,#mermaid-svg-76LTDhLqxUxVTDcj .image-shape .label,#mermaid-svg-76LTDhLqxUxVTDcj .icon-shape .label{text-align:center;}#mermaid-svg-76LTDhLqxUxVTDcj .node.clickable{cursor:pointer;}#mermaid-svg-76LTDhLqxUxVTDcj .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-76LTDhLqxUxVTDcj .arrowheadPath{fill:#333333;}#mermaid-svg-76LTDhLqxUxVTDcj .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-76LTDhLqxUxVTDcj .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-76LTDhLqxUxVTDcj .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-76LTDhLqxUxVTDcj .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-76LTDhLqxUxVTDcj .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-76LTDhLqxUxVTDcj .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-76LTDhLqxUxVTDcj .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-76LTDhLqxUxVTDcj .cluster text{fill:#333;}#mermaid-svg-76LTDhLqxUxVTDcj .cluster span{color:#333;}#mermaid-svg-76LTDhLqxUxVTDcj div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-76LTDhLqxUxVTDcj .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-76LTDhLqxUxVTDcj rect.text{fill:none;stroke-width:0;}#mermaid-svg-76LTDhLqxUxVTDcj .icon-shape,#mermaid-svg-76LTDhLqxUxVTDcj .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-76LTDhLqxUxVTDcj .icon-shape p,#mermaid-svg-76LTDhLqxUxVTDcj .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-76LTDhLqxUxVTDcj .icon-shape .label rect,#mermaid-svg-76LTDhLqxUxVTDcj .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-76LTDhLqxUxVTDcj .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-76LTDhLqxUxVTDcj .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-76LTDhLqxUxVTDcj :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

I/O密集

CPU密集

批量处理场景

选择并发模型

多线程

多进程

断点续传

结果校验

自动报告

8.2 常见问题FAQ

Q1：线程数开得越多越快吗？ A：不是。对于磁盘I/O，超过一定数量（通常2~4倍CPU核心数）会导致磁盘寻道时间增加，反而变慢。建议测试不同线程数。

Q2：处理OCR时多进程内存爆炸怎么办？ A：减少进程数，或使用multiprocessing的maxtasksperchild参数定期重启进程释放内存。

Q3：断点续传的进度文件会不会越来越大？ A：SQLite数据库存储10万条记录也仅有几十MB，不用担心。JSON方式在文件数超过10万时加载变慢，建议使用SQLite。

Q4：如何让脚本支持不同的PDF模板（发票、合同混批）？ A：可以先根据文件名关键字或PDF第一页内容判断模板类型，然后动态调用对应的提取函数。

8.3 下一步优化方向

• 分布式处理：当单机处理速度不足时，可以使用Celery + Redis将任务分发到多台机器。
• 实时监控：集成Prometheus指标，在Grafana展示处理速率、失败率。
• 智能重试：对于网络错误等瞬时故障，使用指数退避重试。

最后：批量处理PDF看似简单，但要做成健壮、可观测、可恢复的工程，需要综合运用并发、持久化、日志、报告等技术。希望本文提供的代码模板能成为你自动化办公的“瑞士军刀”。如果你在实际使用中遇到新的坑，欢迎留言讨论。

原创不易，如果本文对你有帮助，请点赞、收藏、转发支持！