Python反反爬实战：Playwright+指纹抹除绕过网站检测的6个技巧

Python反反爬实战：Playwright+指纹抹除绕过网站检测的6个技巧

前言做量化数据采集，绕不开一个问题：反爬。东财、同花顺、新浪财经……主流财经网站的反爬机制一个比一个狠。用requests直接请求？分分钟被封IP。用Selenium？检测到webdriver=true直接返回空数据。本文分享我在搭建A股数据采集系统时，用Playwright+指纹抹除绕过网站检测的6个实战技巧。所有代码均在Python 3.10+环境下验证通过。## 一、为什么选Playwright而不是Selenium？先说结论：Selenium已不适合做数据采集。| 维度 | Selenium | Playwright ||——|———-|————|| 指纹检测 | 容易被识别 | 原生更干净 || 速度 | 较慢 | 快2-3倍 || 网络拦截 | 不支持 | 原生支持 || 异步支持 | 差 | 原生async || 反检测插件 | 需要额外配置 | 生态更成熟 |关键原因：Selenium会注入window.navigator.webdriver = true，几乎所有主流反爬系统都会检测这个字段。Playwright虽然也会注入，但通过playwright-stealth插件可以轻松抹除。## 二、环境搭建bashpip install playwright playwright-stealthplaywright install chromium核心依赖就两个：- playwright：浏览器自动化框架- playwright-stealth：指纹抹除插件，一站式解决webdriver、chrome属性、语言属性等检测点## 三、6个实战技巧### 技巧1：基础指纹抹除这是最基本的操作，抹除Playwright的自动化特征：pythonfrom playwright.async_api import async_playwrightfrom playwright_stealth import stealth_asyncasync def create_stealth_browser(): pw = await async_playwright().start() browser = await pw.chromium.launch(headless=False) context = await browser.new_context( viewport={"width": 1920, "height": 1080}, user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" ) page = await context.new_page() # 关键：注入stealth脚本 await stealth_async(page) return page````stealth_async(page)` 一行代码搞定以下检测点：- `navigator.webdriver` 设为 undefined- 伪造 `chrome.runtime` 属性- 补全 `navigator.plugins` 数组- 伪造 `navigator.languages`- 修复 `window.chrome` 加载时序### 技巧2：随机化User-Agent固定UA是最容易被检测的点。建议维护一个UA池，每次请求随机选取：pythonimport randomUA_POOL = [ “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36”, “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36”, “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36”, “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0”,]context = await browser.new_context( user_agent=random.choice(UA_POOL),)**踩坑记录**：UA要和浏览器内核匹配。用Chrome的UA但实际跑Firefox内核，会被`navigator.userAgentData`检测出来。### 技巧3：Viewport和屏幕分辨率一致化很多反爬系统会检查`window.screen`和`viewport`是否一致：pythonimport random# 常见分辨率池VIEWPORTS = [ {“width”: 1920, “height”: 1080}, {“width”: 1366, “height”: 768}, {“width”: 1536, “height”: 864}, {“width”: 1440, “height”: 900},]viewport = random.choice(VIEWPORTS)context = await browser.new_context( viewport=viewport, screen=viewport, # 关键：screen和viewport保持一致)**踩坑记录**：如果viewport设为1920×1080但screen是1366×768，部分反爬脚本会判定为自动化工具。### 技巧4：请求间隔随机化+行为模拟固定间隔请求是最典型的爬虫特征。模拟人类行为的关键是**随机化**：pythonimport asyncioimport randomasync def human_like_delay(): “”“模拟人类操作间隔”“” await asyncio.sleep(random.uniform(0.5, 2.5))async def random_scroll(page): “”“模拟人类随机滚动”“” for _ in range(random.randint(1, 3)): scroll_y = random.randint(100, 500) await page.evaluate(f"window.scrollBy(0, {scroll_y})“) await asyncio.sleep(random.uniform(0.3, 1.0))async def human_click(page, selector): “”“模拟人类点击（带偏移）””" element = await page.query_selector(selector) box = await element.bounding_box() if box: # 点击位置加随机偏移 x = box[“x”] + box[“width”] * random.uniform(0.2, 0.8) y = box[“y”] + box[“height”] * random.uniform(0.2, 0.8) await page.mouse.click(x, y) await asyncio.sleep(random.uniform(0.3, 0.8))### 技巧5：拦截请求屏蔽无用资源加载图片、CSS、字体等资源不仅拖慢速度，还会增加被检测的概率（资源加载时序异常）：pythonasync def block_unnecessary_resources(route): “”“屏蔽非必要资源请求”“” if route.request.resource_type in [“image”, “stylesheet”, “font”, “media”]: await route.abort() else: await route.continue_()await page.route(“/*“, block_unnecessary_resources)这一招能把页面加载时间从5秒降到1秒以内，同时减少网络指纹特征。### 技巧6：Cookie持久化+会话复用频繁创建新会话是爬虫的典型特征。正确做法是复用Cookie：pythonimport jsonimport osCOOKIE_FILE = “cookies.json"async def save_cookies(context): “”“保存cookie到本地””” cookies = await context.cookies() with open(COOKIE_FILE, “w”) as f: json.dump(cookies, f)async def load_cookies(context): “”“从本地加载cookie”“” if os.path.exists(COOKIE_FILE): with open(COOKIE_FILE, “r”) as f: cookies = json.load(f) await context.add_cookies(cookies) return True return False# 使用示例async def main(): page = await create_stealth_browser() context = page.context # 尝试复用已有cookie if not await load_cookies(context): # 首次访问，正常浏览获取cookie await page.goto(“https://example.com”) await human_like_delay() await save_cookies(context) # 后续请求直接复用 await page.goto(“https://example.com/data”)**踩坑记录**：Cookie有过期时间，建议每次请求后检查是否还有效。东财的Cookie有效期通常是2小时，超过后需要重新获取。## 四、完整示例：采集东财个股资金流把上面的技巧组合起来，采集东方财富个股资金流数据：pythonimport asyncioimport jsonfrom playwright.async_api import async_playwrightfrom playwright_stealth import stealth_asyncasync def fetch_money_flow(stock_code=“000001”): “”“采集个股资金流数据”“” async with async_playwright() as pw: browser = await pw.chromium.launch(headless=True) context = await browser.new_context( viewport={“width”: 1920, “height”: 1080}, screen={“width”: 1920, “height”: 1080}, user_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36” ) page = await context.new_page() await stealth_async(page) # 屏蔽无用资源 await page.route("/*”, lambda route: ( route.abort() if route.request.resource_type in [“image”, “stylesheet”, “font”] else route.continue_() )) # 访问资金流页面 url = f"https://data.eastmoney.com/zjlx/{stock_code}.html" await page.goto(url, wait_until=“networkidle”) # 等待数据加载 await page.wait_for_selector(“.flow-data”, timeout=10000) # 提取数据 data = await page.evaluate(“”" () => { const rows = document.querySelectorAll(‘.flow-data tr’); return Array.from(rows).map(row => Array.from(row.cells).map(cell => cell.textContent.trim()) ); } “”") await browser.close() return data# 运行result = asyncio.run(fetch_money_flow(“000001”))print(json.dumps(result, ensure_ascii=False, indent=2))```## 五、注意事项1. 合规使用：采集频率控制在合理范围，建议单IP每分钟不超过10次请求2. IP代理：大规模采集时建议配合代理池，推荐按次计费的短效代理3. 验证码：遇到验证码说明已被标记，需要更换IP或降低频率4. 数据时效：网站DOM结构可能更新，选择器需要定期维护## 六、总结反爬对抗本质是指纹博弈。核心思路就三条：1. 抹除自动化特征：stealth插件一键搞定2. 模拟人类行为：随机化一切可随机参数3. 减少请求特征：屏蔽无用资源、复用Cookie这套方案在我实际运行的数据采集系统中稳定运行了3个月，日均请求5000+次，被封率低于0.1%。—作者简介：h29hj，A股量化交易者，专注Python数据采集与量化分析。欢迎关注获取更多实战分享。