Python Scrapy 爬虫框架：分布式数据采集实战指南-171主机测评

Python Scrapy 爬虫框架：分布式数据采集实战指南

Scrapy 是 Python 生态中最强大的爬虫框架，它异步、可扩展、内置数据管道。本文将带你从单站点爬虫到分布式集群，掌握生产级数据采集的完整技术栈。

一、Scrapy 核心架构

Engine ──▶ Scheduler ──▶ Downloader ──▶ Spiders ──▶ Item Pipeline
│ │ │ │ │
└───────────┴──────────────┴─────────────┴──────────────┘
数据流

二、快速开始

pip install scrapy
scrapy startproject ecommerce_crawler
cd ecommerce_crawler

基础 Spider

# spiders/product_spider.py
import scrapy
from ecommerce_crawler.items import ProductItem

class ProductSpider(scrapy.Spider):
name = 'products'
allowed_domains = ['example-shop.com']
start_urls = ['https://example-shop.com/products']

custom_settings = {
'DOWNLOAD_DELAY': 1,
'CONCURRENT_REQUESTS': 16,
'RETRY_TIMES': 3,
'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}

def parse(self, response):
"""解析商品列表页"""
products = response.css('div.product-card')

for product in products:
item = ProductItem()
item['name'] = product.css('h2.title::text').get(default='').strip()
item['price'] = self.extract_price(product.css('span.price::text').get())
item['url'] = response.urljoin(product.css('a::attr(href)').get())
item['category'] = response.css('h1.category-title::text').get()

# 进入详情页
yield scrapy.Request(
url=item['url'],
callback=self.parse_detail,
meta={'item': item}
)

# 分页
next_page = response.css('a.next-page::attr(href)').get()
if next_page:
yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

def parse_detail(self, response):
"""解析商品详情页"""
item = response.meta['item']
item['description'] = response.css('div.description::text').get(default='')
item['sku'] = response.css('span.sku::text').get()
item['stock'] = response.css('span.stock::text').get()
item['images'] = response.css('img.product-image::attr(src)').getall()
item['crawl_time'] = datetime.now().isoformat()

yield item

def extract_price(self, price_text):
"""提取价格数值"""
if not price_text:
return None
match = re.search(r'[\\d,]+\\.?\\d*', price_text.replace(',', ''))
return float(match.group()) if match else None

Item 定义

# i

Python Scrapy 爬虫框架：分布式数据采集实战指南