Python爬虫速度慢到离谱？用这招让数据抓取快10倍

你有没有遇到过这种情况：写好的Python爬虫脚本，跑1000条数据要花2个小时，换了更贵的代理、加了更短的延迟，速度还是提不上来，甚至还因为请求太密集被封了IP？

其实绝大多数Python爬虫慢，根本不是硬件或者代理的问题，而是你用了串行请求的方式——一条请求发出去，等服务器返回结果，再发下一条，中间大量的时间都浪费在等待上了。

今天就给你分享一个工业级的Python爬虫提速方案：异步协程+多线程混合架构，配合连接池复用和动态请求调度，实测能让数据抓取速度提升8-12倍，同时还能降低被封IP的风险。

一、先搞懂：你的爬虫为什么慢？

我们先看一下串行请求的时间线，用一个简单的例子说明：

假设你要爬100条数据，每条数据的请求+响应时间是1秒，串行请求的总耗时就是100秒，中间99秒的时间，CPU和网络都在空等，完全没有被利用起来。

#mermaid-svg-ZOzkBDZrK46piHL6{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ZOzkBDZrK46piHL6 .error-icon{fill:#552222;}#mermaid-svg-ZOzkBDZrK46piHL6 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ZOzkBDZrK46piHL6 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ZOzkBDZrK46piHL6 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ZOzkBDZrK46piHL6 .marker.cross{stroke:#333333;}#mermaid-svg-ZOzkBDZrK46piHL6 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ZOzkBDZrK46piHL6 p{margin:0;}#mermaid-svg-ZOzkBDZrK46piHL6 .mermaid-main-font{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-ZOzkBDZrK46piHL6 .exclude-range{fill:#eeeeee;}#mermaid-svg-ZOzkBDZrK46piHL6 .section{stroke:none;opacity:0.2;}#mermaid-svg-ZOzkBDZrK46piHL6 .section0{fill:rgba(102, 102, 255, 0.49);}#mermaid-svg-ZOzkBDZrK46piHL6 .section2{fill:#fff400;}#mermaid-svg-ZOzkBDZrK46piHL6 .section1,#mermaid-svg-ZOzkBDZrK46piHL6 .section3{fill:white;opacity:0.2;}#mermaid-svg-ZOzkBDZrK46piHL6 .sectionTitle0{fill:#333;}#mermaid-svg-ZOzkBDZrK46piHL6 .sectionTitle1{fill:#333;}#mermaid-svg-ZOzkBDZrK46piHL6 .sectionTitle2{fill:#333;}#mermaid-svg-ZOzkBDZrK46piHL6 .sectionTitle3{fill:#333;}#mermaid-svg-ZOzkBDZrK46piHL6 .sectionTitle{text-anchor:start;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-ZOzkBDZrK46piHL6 .grid .tick{stroke:lightgrey;opacity:0.8;shape-rendering:crispEdges;}#mermaid-svg-ZOzkBDZrK46piHL6 .grid .tick text{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;fill:#333;}#mermaid-svg-ZOzkBDZrK46piHL6 .grid path{stroke-width:0;}#mermaid-svg-ZOzkBDZrK46piHL6 .today{fill:none;stroke:red;stroke-width:2px;}#mermaid-svg-ZOzkBDZrK46piHL6 .task{stroke-width:2;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskText{text-anchor:middle;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutsideRight{fill:black;text-anchor:start;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutsideLeft{fill:black;text-anchor:end;}#mermaid-svg-ZOzkBDZrK46piHL6 .task.clickable{cursor:pointer;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskText.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskText0,#mermaid-svg-ZOzkBDZrK46piHL6 .taskText1,#mermaid-svg-ZOzkBDZrK46piHL6 .taskText2,#mermaid-svg-ZOzkBDZrK46piHL6 .taskText3{fill:white;}#mermaid-svg-ZOzkBDZrK46piHL6 .task0,#mermaid-svg-ZOzkBDZrK46piHL6 .task1,#mermaid-svg-ZOzkBDZrK46piHL6 .task2,#mermaid-svg-ZOzkBDZrK46piHL6 .task3{fill:#8a90dd;stroke:#534fbc;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutside0,#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutside2{fill:black;}#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutside1,#mermaid-svg-ZOzkBDZrK46piHL6 .taskTextOutside3{fill:black;}#mermaid-svg-ZOzkBDZrK46piHL6 .active0,#mermaid-svg-ZOzkBDZrK46piHL6 .active1,#mermaid-svg-ZOzkBDZrK46piHL6 .active2,#mermaid-svg-ZOzkBDZrK46piHL6 .active3{fill:#bfc7ff;stroke:#534fbc;}#mermaid-svg-ZOzkBDZrK46piHL6 .activeText0,#mermaid-svg-ZOzkBDZrK46piHL6 .activeText1,#mermaid-svg-ZOzkBDZrK46piHL6 .activeText2,#mermaid-svg-ZOzkBDZrK46piHL6 .activeText3{fill:black!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .done0,#mermaid-svg-ZOzkBDZrK46piHL6 .done1,#mermaid-svg-ZOzkBDZrK46piHL6 .done2,#mermaid-svg-ZOzkBDZrK46piHL6 .done3{stroke:grey;fill:lightgrey;stroke-width:2;}#mermaid-svg-ZOzkBDZrK46piHL6 .doneText0,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText1,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText2,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText3{fill:black!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .doneText0.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText0.taskTextOutsideRight,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText1.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText1.taskTextOutsideRight,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText2.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText2.taskTextOutsideRight,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText3.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .crit0,#mermaid-svg-ZOzkBDZrK46piHL6 .crit1,#mermaid-svg-ZOzkBDZrK46piHL6 .crit2,#mermaid-svg-ZOzkBDZrK46piHL6 .crit3{stroke:#ff8888;fill:red;stroke-width:2;}#mermaid-svg-ZOzkBDZrK46piHL6 .activeCrit0,#mermaid-svg-ZOzkBDZrK46piHL6 .activeCrit1,#mermaid-svg-ZOzkBDZrK46piHL6 .activeCrit2,#mermaid-svg-ZOzkBDZrK46piHL6 .activeCrit3{stroke:#ff8888;fill:#bfc7ff;stroke-width:2;}#mermaid-svg-ZOzkBDZrK46piHL6 .doneCrit0,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCrit1,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCrit2,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCrit3{stroke:#ff8888;fill:lightgrey;stroke-width:2;cursor:pointer;shape-rendering:crispEdges;}#mermaid-svg-ZOzkBDZrK46piHL6 .milestone{transform:rotate(45deg) scale(0.8,0.8);}#mermaid-svg-ZOzkBDZrK46piHL6 .milestoneText{font-style:italic;}#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText0,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText1,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText2,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText3{fill:black!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText0.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText0.taskTextOutsideRight,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText1.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText1.taskTextOutsideRight,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText2.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText2.taskTextOutsideRight,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText3.taskTextOutsideLeft,#mermaid-svg-ZOzkBDZrK46piHL6 .doneCritText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .vert{stroke:navy;}#mermaid-svg-ZOzkBDZrK46piHL6 .vertText{font-size:15px;text-anchor:middle;fill:navy!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .activeCritText0,#mermaid-svg-ZOzkBDZrK46piHL6 .activeCritText1,#mermaid-svg-ZOzkBDZrK46piHL6 .activeCritText2,#mermaid-svg-ZOzkBDZrK46piHL6 .activeCritText3{fill:black!important;}#mermaid-svg-ZOzkBDZrK46piHL6 .titleText{text-anchor:middle;font-size:18px;fill:#333;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}#mermaid-svg-ZOzkBDZrK46piHL6 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;} 00 15 30 45 00 15 30请求1 请求2 请求3 … 请求100 串行请求串行请求时间线

而如果用异步协程+多线程混合架构，我们可以同时发起几十甚至上百条请求，CPU和网络全程满负荷运行，总耗时只取决于最慢的那几条请求，100条数据可能只需要10秒就能爬完。

二、核心方案：异步协程+多线程混合架构

为什么要用混合架构，而不是纯异步或者纯多线程？

• 纯异步协程：适合IO密集型任务（比如网络请求），但Python的GIL锁会限制CPU密集型任务（比如数据解析、图片处理）的性能；
• 纯多线程：虽然能绕过部分GIL锁的限制，但线程切换开销大，同时发起的请求数量有限；
• 异步协程+多线程混合架构：用异步协程处理网络请求（IO密集型），用多线程处理数据解析（CPU密集型），两者优势互补，性能最大化。

我们来看一下混合架构的系统流程图：

#mermaid-svg-55TP7isRnfWu6blD{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-55TP7isRnfWu6blD .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-55TP7isRnfWu6blD .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-55TP7isRnfWu6blD .error-icon{fill:#552222;}#mermaid-svg-55TP7isRnfWu6blD .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-55TP7isRnfWu6blD .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-55TP7isRnfWu6blD .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-55TP7isRnfWu6blD .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-55TP7isRnfWu6blD .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-55TP7isRnfWu6blD .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-55TP7isRnfWu6blD .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-55TP7isRnfWu6blD .marker{fill:#333333;stroke:#333333;}#mermaid-svg-55TP7isRnfWu6blD .marker.cross{stroke:#333333;}#mermaid-svg-55TP7isRnfWu6blD svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-55TP7isRnfWu6blD p{margin:0;}#mermaid-svg-55TP7isRnfWu6blD .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-55TP7isRnfWu6blD .cluster-label text{fill:#333;}#mermaid-svg-55TP7isRnfWu6blD .cluster-label span{color:#333;}#mermaid-svg-55TP7isRnfWu6blD .cluster-label span p{background-color:transparent;}#mermaid-svg-55TP7isRnfWu6blD .label text,#mermaid-svg-55TP7isRnfWu6blD span{fill:#333;color:#333;}#mermaid-svg-55TP7isRnfWu6blD .node rect,#mermaid-svg-55TP7isRnfWu6blD .node circle,#mermaid-svg-55TP7isRnfWu6blD .node ellipse,#mermaid-svg-55TP7isRnfWu6blD .node polygon,#mermaid-svg-55TP7isRnfWu6blD .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-55TP7isRnfWu6blD .rough-node .label text,#mermaid-svg-55TP7isRnfWu6blD .node .label text,#mermaid-svg-55TP7isRnfWu6blD .image-shape .label,#mermaid-svg-55TP7isRnfWu6blD .icon-shape .label{text-anchor:middle;}#mermaid-svg-55TP7isRnfWu6blD .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-55TP7isRnfWu6blD .rough-node .label,#mermaid-svg-55TP7isRnfWu6blD .node .label,#mermaid-svg-55TP7isRnfWu6blD .image-shape .label,#mermaid-svg-55TP7isRnfWu6blD .icon-shape .label{text-align:center;}#mermaid-svg-55TP7isRnfWu6blD .node.clickable{cursor:pointer;}#mermaid-svg-55TP7isRnfWu6blD .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-55TP7isRnfWu6blD .arrowheadPath{fill:#333333;}#mermaid-svg-55TP7isRnfWu6blD .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-55TP7isRnfWu6blD .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-55TP7isRnfWu6blD .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-55TP7isRnfWu6blD .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-55TP7isRnfWu6blD .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-55TP7isRnfWu6blD .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-55TP7isRnfWu6blD .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-55TP7isRnfWu6blD .cluster text{fill:#333;}#mermaid-svg-55TP7isRnfWu6blD .cluster span{color:#333;}#mermaid-svg-55TP7isRnfWu6blD div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-55TP7isRnfWu6blD .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-55TP7isRnfWu6blD rect.text{fill:none;stroke-width:0;}#mermaid-svg-55TP7isRnfWu6blD .icon-shape,#mermaid-svg-55TP7isRnfWu6blD .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-55TP7isRnfWu6blD .icon-shape p,#mermaid-svg-55TP7isRnfWu6blD .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-55TP7isRnfWu6blD .icon-shape .label rect,#mermaid-svg-55TP7isRnfWu6blD .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-55TP7isRnfWu6blD .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-55TP7isRnfWu6blD .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-55TP7isRnfWu6blD :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

任务队列

异步协程调度器

异步HTTP连接池

发起批量网络请求

获取响应数据

多线程解析队列

多线程数据解析

数据存储

三、工业级代码实现

下面我们用aiohttp（异步HTTP库）和concurrent.futures.ThreadPoolExecutor（多线程池），实现一个完整的异步协程+多线程混合爬虫，代码可直接复用。

3.1 环境准备

首先安装需要的库：

pip install aiohttp beautifulsoup4 lxml

3.2 完整代码

import asyncio
import aiohttp
from aiohttp import TCPConnector
from concurrent.futures import ThreadPoolExecutor
from bs4 import BeautifulSoup
import random
import time
from fake_useragent import UserAgent

# 全局配置
MAX_CONCURRENT_REQUESTS = 50 # 最大并发请求数，根据目标网站的反爬强度调整
MAX_THREADS = 8 # 最大解析线程数，根据CPU核心数调整
ua = UserAgent(browsers=["chrome", "edge"])

# 模拟数据解析函数（CPU密集型）
def parse_html(html, url):
"""
解析HTML，提取目标数据
这里用BeautifulSoup做示例，实际项目中可以用lxml、正则表达式等
"""
soup = BeautifulSoup(html, "lxml")
# 模拟提取标题
title = soup.title.string.strip() if soup.title else "无标题"
# 模拟提取正文前100个字符
content = soup.get_text().strip()[:100] if soup.get_text() else "无内容"
# 模拟耗时的解析操作（比如图片下载、正则匹配复杂内容）
time.sleep(random.uniform(0.01, 0.05))
return {"url": url, "title": title, "content": content}

# 异步请求函数（IO密集型）
async def fetch_url(session, url, executor):
"""
发起异步HTTP请求，获取响应后提交到多线程池解析
"""
headers = {
"User-Agent": ua.random,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9",
"Connection": "keep-alive"
}
try:
# 发起异步请求，设置超时
async with session.get(url, headers=headers, timeout=10) as response:
if response.status == 200:
html = await response.text()
# 提交到多线程池解析，避免阻塞异步协程
loop = asyncio.get_running_loop()
data = await loop.run_in_executor(executor, parse_html, html, url)
print(f"成功爬取：{url}")
return data
else:
print(f"请求失败，状态码：{response.status}，URL：{url}")
return None
except Exception as e:
print(f"请求异常：{e}，URL：{url}")
return None

# 主调度函数
async def main(urls):
"""
主调度函数：创建异步连接池、多线程池，调度所有请求
"""
# 创建TCP连接池，复用连接，减少握手开销
connector = TCPConnector(limit=MAX_CONCURRENT_REQUESTS, limit_per_host=10)
# 创建多线程池
executor = ThreadPoolExecutor(max_workers=MAX_THREADS)

async with aiohttp.ClientSession(connector=connector) as session:
# 创建所有请求的协程任务
tasks = [fetch_url(session, url, executor) for url in urls]
# 并发执行所有任务
results = await asyncio.gather(*tasks, return_exceptions=False)

# 关闭多线程池
executor.shutdown(wait=True)
# 过滤掉失败的结果
valid_results = [r for r in results if r is not None]
print(f"\\n爬取完成！共爬取{len(valid_results)}条有效数据")
return valid_results

if __name__ == "__main__":
# 生成测试URL（这里用httpbin模拟，实际项目中替换为你的目标URL）
test_urls = [f"https://httpbin.org/html?i={i}" for i in range(1000)]

# 测试串行请求的耗时（注释掉混合架构的代码，取消下面的注释即可测试）
# import requests
# start_time = time.time()
# for url in test_urls[:100]: # 只测100条，不然太慢
# headers = {"User-Agent": ua.random}
# response = requests.get(url, headers=headers, timeout=10)
# parse_html(response.text, url)
# serial_time = time.time() – start_time
# print(f"串行请求100条数据耗时：{serial_time:.2f}秒")

# 测试混合架构的耗时
start_time = time.time()
asyncio.run(main(test_urls))
mixed_time = time.time() – start_time
print(f"混合架构爬取1000条数据耗时：{mixed_time:.2f}秒")

四、关键优化点解析

4.1 异步HTTP连接池复用

用aiohttp.TCPConnector创建连接池，设置limit（最大总连接数）和limit_per_host（每个域名的最大连接数），复用TCP连接，减少三次握手和四次挥手的开销，这是异步爬虫提速的核心。

4.2 动态并发数调整

不要把MAX_CONCURRENT_REQUESTS设得太大，否则会因为请求太密集被封IP，也不要设得太小，浪费性能。建议从10开始逐步上调，直到目标网站的响应时间开始变长，或者出现429（请求过多）状态码，然后再下调10%-20%。

4.3 多线程解析数据

用concurrent.futures.ThreadPoolExecutor创建多线程池，把数据解析、图片处理等CPU密集型任务提交到多线程池执行，避免阻塞异步协程，让网络请求和数据解析并行进行，性能最大化。

4.4 随机延迟与UA轮换

虽然混合架构速度快，但还是要加随机延迟和UA轮换，降低被封IP的风险。可以在fetch_url函数中，请求前加一个await asyncio.sleep(random.uniform(0.1, 0.3))的小延迟。

五、实测效果对比

我们用上面的代码，在相同的硬件环境（Intel i5-10400 CPU、16GB内存、100Mbps宽带）下，测试了串行请求和混合架构的耗时：

可以看到，混合架构的速度提升了11.5倍，完全符合我们的预期。

六、避坑指南