Playwright 与 Midscene 原理&安装讲解

Playwright 与 Midscene：AI 定位原理完全指南

本文详细讲解 Midscene 作为 AI 定位框架的原理、与 Playwright 的集成方式、使用方式、操作手段与具象性，并通过流程图帮助读者建立完整认知。

一、核心概念：什么是 Midscene？

Midscene.js 是由字节跳动 Web Infra 团队开源的 AI 驱动、视觉感知 UI 自动化框架，已发布 v1.0 版本。其核心定位是：

采用纯视觉（Pure Vision）路线完成元素定位：基于截图完成所有分析，无需 DOM 信息即可进行 UI 操作。

1.1 Midscene 与 Playwright 的关系

简单理解：Playwright 是「手」，Midscene 是「眼睛和大脑」。Midscene 通过 AI 视觉模型分析截图，告诉 Playwright「点哪里」「输入什么」。

1.2 技术路线对比

1.3 Midscene 支持平台

• Web：与 Playwright、Puppeteer 集成，或桥接模式控制桌面浏览器
• iOS：配合 WebDriverAgent
• Android：配合 adb
• 任意界面：通过 JavaScript SDK 控制自定义 GUI

二、AI 定位的核心原理

2.1 纯视觉路线（Pure Vision）

Midscene 从 v1.0 起只支持纯视觉方案，不再提供 DOM 提取兼容模式。

核心流程：

2.2 为什么选择纯视觉？

根据 Midscene 官方文档（模型策略），经过大量项目验证后：

2.3 视觉定位流程图

#mermaid-svg-rwOFuKdZLZPAnAHC{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-rwOFuKdZLZPAnAHC .error-icon{fill:#552222;}#mermaid-svg-rwOFuKdZLZPAnAHC .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-rwOFuKdZLZPAnAHC .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-rwOFuKdZLZPAnAHC .marker{fill:#333333;stroke:#333333;}#mermaid-svg-rwOFuKdZLZPAnAHC .marker.cross{stroke:#333333;}#mermaid-svg-rwOFuKdZLZPAnAHC svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-rwOFuKdZLZPAnAHC p{margin:0;}#mermaid-svg-rwOFuKdZLZPAnAHC .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC .cluster-label text{fill:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC .cluster-label span{color:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC .cluster-label span p{background-color:transparent;}#mermaid-svg-rwOFuKdZLZPAnAHC .label text,#mermaid-svg-rwOFuKdZLZPAnAHC span{fill:#333;color:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC .node rect,#mermaid-svg-rwOFuKdZLZPAnAHC .node circle,#mermaid-svg-rwOFuKdZLZPAnAHC .node ellipse,#mermaid-svg-rwOFuKdZLZPAnAHC .node polygon,#mermaid-svg-rwOFuKdZLZPAnAHC .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-rwOFuKdZLZPAnAHC .rough-node .label text,#mermaid-svg-rwOFuKdZLZPAnAHC .node .label text,#mermaid-svg-rwOFuKdZLZPAnAHC .image-shape .label,#mermaid-svg-rwOFuKdZLZPAnAHC .icon-shape .label{text-anchor:middle;}#mermaid-svg-rwOFuKdZLZPAnAHC .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-rwOFuKdZLZPAnAHC .rough-node .label,#mermaid-svg-rwOFuKdZLZPAnAHC .node .label,#mermaid-svg-rwOFuKdZLZPAnAHC .image-shape .label,#mermaid-svg-rwOFuKdZLZPAnAHC .icon-shape .label{text-align:center;}#mermaid-svg-rwOFuKdZLZPAnAHC .node.clickable{cursor:pointer;}#mermaid-svg-rwOFuKdZLZPAnAHC .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-rwOFuKdZLZPAnAHC .arrowheadPath{fill:#333333;}#mermaid-svg-rwOFuKdZLZPAnAHC .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-rwOFuKdZLZPAnAHC .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-rwOFuKdZLZPAnAHC .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-rwOFuKdZLZPAnAHC .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-rwOFuKdZLZPAnAHC .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-rwOFuKdZLZPAnAHC .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-rwOFuKdZLZPAnAHC .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-rwOFuKdZLZPAnAHC .cluster text{fill:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC .cluster span{color:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-rwOFuKdZLZPAnAHC .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-rwOFuKdZLZPAnAHC rect.text{fill:none;stroke-width:0;}#mermaid-svg-rwOFuKdZLZPAnAHC .icon-shape,#mermaid-svg-rwOFuKdZLZPAnAHC .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-rwOFuKdZLZPAnAHC .icon-shape p,#mermaid-svg-rwOFuKdZLZPAnAHC .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-rwOFuKdZLZPAnAHC .icon-shape rect,#mermaid-svg-rwOFuKdZLZPAnAHC .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-rwOFuKdZLZPAnAHC .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-rwOFuKdZLZPAnAHC .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-rwOFuKdZLZPAnAHC :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

是

否

开始

获取当前页面截图

截图 + 用户指令送入 VL 模型

VL 模型分析图像并识别目标

定位成功?

返回元素坐标 center/bbox

抛出错误

Playwright 根据坐标执行 click/type

操作完成

结束

2.4 支持的视觉模型

Midscene 要求模型具备 Visual Grounding（视觉定位） 能力，推荐使用：

注意：GPT-5 等多模态模型在 UI 定位（Visual Grounding） 场景表现较差，不适合作为 Midscene 的默认模型。

三、使用方式

3.1 安装与配置

npm install @midscene/web playwright @playwright/test tsx –save-dev

配置环境变量（以千问为例）：

export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export MIDSCENE_MODEL_API_KEY="你的 API Key"
export MIDSCENE_MODEL_NAME="qwen3-vl-plus"
export MIDSCENE_MODEL_FAMILY="qwen3-vl"

3.2 两种集成方式

方式一：直接脚本集成（适合快速体验、数据抓取）

import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PlaywrightAgent(page);

// 自然语言执行操作
await agent.aiAct('type "Headphones" in search box, hit Enter');
await agent.aiWaitFor('there is at least one headphone item on page');

// 数据提取
const items = await agent.aiQuery(
'{itemTitle: string, price: Number}[], find item in list and corresponding price'
);
console.log(items);

await browser.close();

方式二：Playwright 测试用例集成（适合 E2E 测试）

在 playwright.config.ts 中配置 reporter，并扩展 test fixture，即可在测试中直接使用 aiTap、aiInput、aiQuery 等。

3.3 完整交互流程图

#mermaid-svg-PBmDgTf25b0BCIpU{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-PBmDgTf25b0BCIpU .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-PBmDgTf25b0BCIpU .error-icon{fill:#552222;}#mermaid-svg-PBmDgTf25b0BCIpU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-PBmDgTf25b0BCIpU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-PBmDgTf25b0BCIpU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-PBmDgTf25b0BCIpU .marker.cross{stroke:#333333;}#mermaid-svg-PBmDgTf25b0BCIpU svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-PBmDgTf25b0BCIpU p{margin:0;}#mermaid-svg-PBmDgTf25b0BCIpU .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-PBmDgTf25b0BCIpU .cluster-label text{fill:#333;}#mermaid-svg-PBmDgTf25b0BCIpU .cluster-label span{color:#333;}#mermaid-svg-PBmDgTf25b0BCIpU .cluster-label span p{background-color:transparent;}#mermaid-svg-PBmDgTf25b0BCIpU .label text,#mermaid-svg-PBmDgTf25b0BCIpU span{fill:#333;color:#333;}#mermaid-svg-PBmDgTf25b0BCIpU .node rect,#mermaid-svg-PBmDgTf25b0BCIpU .node circle,#mermaid-svg-PBmDgTf25b0BCIpU .node ellipse,#mermaid-svg-PBmDgTf25b0BCIpU .node polygon,#mermaid-svg-PBmDgTf25b0BCIpU .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-PBmDgTf25b0BCIpU .rough-node .label text,#mermaid-svg-PBmDgTf25b0BCIpU .node .label text,#mermaid-svg-PBmDgTf25b0BCIpU .image-shape .label,#mermaid-svg-PBmDgTf25b0BCIpU .icon-shape .label{text-anchor:middle;}#mermaid-svg-PBmDgTf25b0BCIpU .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-PBmDgTf25b0BCIpU .rough-node .label,#mermaid-svg-PBmDgTf25b0BCIpU .node .label,#mermaid-svg-PBmDgTf25b0BCIpU .image-shape .label,#mermaid-svg-PBmDgTf25b0BCIpU .icon-shape .label{text-align:center;}#mermaid-svg-PBmDgTf25b0BCIpU .node.clickable{cursor:pointer;}#mermaid-svg-PBmDgTf25b0BCIpU .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-PBmDgTf25b0BCIpU .arrowheadPath{fill:#333333;}#mermaid-svg-PBmDgTf25b0BCIpU .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-PBmDgTf25b0BCIpU .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-PBmDgTf25b0BCIpU .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-PBmDgTf25b0BCIpU .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-PBmDgTf25b0BCIpU .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-PBmDgTf25b0BCIpU .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-PBmDgTf25b0BCIpU .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-PBmDgTf25b0BCIpU .cluster text{fill:#333;}#mermaid-svg-PBmDgTf25b0BCIpU .cluster span{color:#333;}#mermaid-svg-PBmDgTf25b0BCIpU div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-PBmDgTf25b0BCIpU .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-PBmDgTf25b0BCIpU rect.text{fill:none;stroke-width:0;}#mermaid-svg-PBmDgTf25b0BCIpU .icon-shape,#mermaid-svg-PBmDgTf25b0BCIpU .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-PBmDgTf25b0BCIpU .icon-shape p,#mermaid-svg-PBmDgTf25b0BCIpU .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-PBmDgTf25b0BCIpU .icon-shape rect,#mermaid-svg-PBmDgTf25b0BCIpU .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-PBmDgTf25b0BCIpU .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-PBmDgTf25b0BCIpU .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-PBmDgTf25b0BCIpU :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

aiAct 自动规划

aiTap/aiInput 即时操作

用户调用 API

操作类型?

Planning 模型拆解为多步骤

直接进入定位

对每步执行视觉定位

截图

VL 模型分析

返回坐标

Playwright 执行 click/type

完成

四、操作手段详解

4.1 API 分类

4.2 自动规划 vs 即时操作

自动规划流程（aiAct）：

#mermaid-svg-qMj5qc5KOb1ATLSZ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-qMj5qc5KOb1ATLSZ .error-icon{fill:#552222;}#mermaid-svg-qMj5qc5KOb1ATLSZ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-qMj5qc5KOb1ATLSZ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .marker.cross{stroke:#333333;}#mermaid-svg-qMj5qc5KOb1ATLSZ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-qMj5qc5KOb1ATLSZ p{margin:0;}#mermaid-svg-qMj5qc5KOb1ATLSZ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .cluster-label text{fill:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .cluster-label span{color:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .cluster-label span p{background-color:transparent;}#mermaid-svg-qMj5qc5KOb1ATLSZ .label text,#mermaid-svg-qMj5qc5KOb1ATLSZ span{fill:#333;color:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .node rect,#mermaid-svg-qMj5qc5KOb1ATLSZ .node circle,#mermaid-svg-qMj5qc5KOb1ATLSZ .node ellipse,#mermaid-svg-qMj5qc5KOb1ATLSZ .node polygon,#mermaid-svg-qMj5qc5KOb1ATLSZ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .rough-node .label text,#mermaid-svg-qMj5qc5KOb1ATLSZ .node .label text,#mermaid-svg-qMj5qc5KOb1ATLSZ .image-shape .label,#mermaid-svg-qMj5qc5KOb1ATLSZ .icon-shape .label{text-anchor:middle;}#mermaid-svg-qMj5qc5KOb1ATLSZ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .rough-node .label,#mermaid-svg-qMj5qc5KOb1ATLSZ .node .label,#mermaid-svg-qMj5qc5KOb1ATLSZ .image-shape .label,#mermaid-svg-qMj5qc5KOb1ATLSZ .icon-shape .label{text-align:center;}#mermaid-svg-qMj5qc5KOb1ATLSZ .node.clickable{cursor:pointer;}#mermaid-svg-qMj5qc5KOb1ATLSZ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .arrowheadPath{fill:#333333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-qMj5qc5KOb1ATLSZ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-qMj5qc5KOb1ATLSZ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-qMj5qc5KOb1ATLSZ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-qMj5qc5KOb1ATLSZ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .cluster text{fill:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ .cluster span{color:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-qMj5qc5KOb1ATLSZ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-qMj5qc5KOb1ATLSZ rect.text{fill:none;stroke-width:0;}#mermaid-svg-qMj5qc5KOb1ATLSZ .icon-shape,#mermaid-svg-qMj5qc5KOb1ATLSZ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-qMj5qc5KOb1ATLSZ .icon-shape p,#mermaid-svg-qMj5qc5KOb1ATLSZ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-qMj5qc5KOb1ATLSZ .icon-shape rect,#mermaid-svg-qMj5qc5KOb1ATLSZ .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-qMj5qc5KOb1ATLSZ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-qMj5qc5KOb1ATLSZ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-qMj5qc5KOb1ATLSZ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

是

否

aiAct 调用

Planning 模型规划步骤

步骤1: 定位目标元素

视觉定位 → 执行

还有下一步?

完成

即时操作流程（aiTap/aiInput）：

#mermaid-svg-dhBbSldCbJLVRb4F{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dhBbSldCbJLVRb4F .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dhBbSldCbJLVRb4F .error-icon{fill:#552222;}#mermaid-svg-dhBbSldCbJLVRb4F .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dhBbSldCbJLVRb4F .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dhBbSldCbJLVRb4F .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dhBbSldCbJLVRb4F .marker.cross{stroke:#333333;}#mermaid-svg-dhBbSldCbJLVRb4F svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dhBbSldCbJLVRb4F p{margin:0;}#mermaid-svg-dhBbSldCbJLVRb4F .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-dhBbSldCbJLVRb4F .cluster-label text{fill:#333;}#mermaid-svg-dhBbSldCbJLVRb4F .cluster-label span{color:#333;}#mermaid-svg-dhBbSldCbJLVRb4F .cluster-label span p{background-color:transparent;}#mermaid-svg-dhBbSldCbJLVRb4F .label text,#mermaid-svg-dhBbSldCbJLVRb4F span{fill:#333;color:#333;}#mermaid-svg-dhBbSldCbJLVRb4F .node rect,#mermaid-svg-dhBbSldCbJLVRb4F .node circle,#mermaid-svg-dhBbSldCbJLVRb4F .node ellipse,#mermaid-svg-dhBbSldCbJLVRb4F .node polygon,#mermaid-svg-dhBbSldCbJLVRb4F .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dhBbSldCbJLVRb4F .rough-node .label text,#mermaid-svg-dhBbSldCbJLVRb4F .node .label text,#mermaid-svg-dhBbSldCbJLVRb4F .image-shape .label,#mermaid-svg-dhBbSldCbJLVRb4F .icon-shape .label{text-anchor:middle;}#mermaid-svg-dhBbSldCbJLVRb4F .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dhBbSldCbJLVRb4F .rough-node .label,#mermaid-svg-dhBbSldCbJLVRb4F .node .label,#mermaid-svg-dhBbSldCbJLVRb4F .image-shape .label,#mermaid-svg-dhBbSldCbJLVRb4F .icon-shape .label{text-align:center;}#mermaid-svg-dhBbSldCbJLVRb4F .node.clickable{cursor:pointer;}#mermaid-svg-dhBbSldCbJLVRb4F .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dhBbSldCbJLVRb4F .arrowheadPath{fill:#333333;}#mermaid-svg-dhBbSldCbJLVRb4F .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dhBbSldCbJLVRb4F .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dhBbSldCbJLVRb4F .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dhBbSldCbJLVRb4F .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dhBbSldCbJLVRb4F .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dhBbSldCbJLVRb4F .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dhBbSldCbJLVRb4F .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dhBbSldCbJLVRb4F .cluster text{fill:#333;}#mermaid-svg-dhBbSldCbJLVRb4F .cluster span{color:#333;}#mermaid-svg-dhBbSldCbJLVRb4F div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dhBbSldCbJLVRb4F .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dhBbSldCbJLVRb4F rect.text{fill:none;stroke-width:0;}#mermaid-svg-dhBbSldCbJLVRb4F .icon-shape,#mermaid-svg-dhBbSldCbJLVRb4F .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dhBbSldCbJLVRb4F .icon-shape p,#mermaid-svg-dhBbSldCbJLVRb4F .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dhBbSldCbJLVRb4F .icon-shape rect,#mermaid-svg-dhBbSldCbJLVRb4F .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dhBbSldCbJLVRb4F .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dhBbSldCbJLVRb4F .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dhBbSldCbJLVRb4F :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

aiTap 调用

截图

VL 模型定位目标

执行 click

完成

建议：明确知道要做什么时，用 aiTap、aiInput 等即时操作；复杂多步任务用 aiAct。

4.3 多模型配合（高阶）

Midscene 支持为不同「意图」配置不同模型：

五、具象性深度解析

5.1 视觉定位的具象性

具象性 指：将「抽象描述」映射为「可执行的具体操作」。

在 Midscene 中：

自然语言："点击登录按钮"
↓ 送入视觉模型（截图 + 指令）
模型输出：{ center: { x: 640, y: 380 }, bbox: {…} }
↓ 坐标映射到实际屏幕
具体操作：page.mouse.click(640, 380)

5.2 具象性保障机制

#mermaid-svg-G59mz7QKbU7TdMoD{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-G59mz7QKbU7TdMoD .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-G59mz7QKbU7TdMoD .error-icon{fill:#552222;}#mermaid-svg-G59mz7QKbU7TdMoD .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-G59mz7QKbU7TdMoD .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-G59mz7QKbU7TdMoD .marker{fill:#333333;stroke:#333333;}#mermaid-svg-G59mz7QKbU7TdMoD .marker.cross{stroke:#333333;}#mermaid-svg-G59mz7QKbU7TdMoD svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-G59mz7QKbU7TdMoD p{margin:0;}#mermaid-svg-G59mz7QKbU7TdMoD .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-G59mz7QKbU7TdMoD .cluster-label text{fill:#333;}#mermaid-svg-G59mz7QKbU7TdMoD .cluster-label span{color:#333;}#mermaid-svg-G59mz7QKbU7TdMoD .cluster-label span p{background-color:transparent;}#mermaid-svg-G59mz7QKbU7TdMoD .label text,#mermaid-svg-G59mz7QKbU7TdMoD span{fill:#333;color:#333;}#mermaid-svg-G59mz7QKbU7TdMoD .node rect,#mermaid-svg-G59mz7QKbU7TdMoD .node circle,#mermaid-svg-G59mz7QKbU7TdMoD .node ellipse,#mermaid-svg-G59mz7QKbU7TdMoD .node polygon,#mermaid-svg-G59mz7QKbU7TdMoD .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-G59mz7QKbU7TdMoD .rough-node .label text,#mermaid-svg-G59mz7QKbU7TdMoD .node .label text,#mermaid-svg-G59mz7QKbU7TdMoD .image-shape .label,#mermaid-svg-G59mz7QKbU7TdMoD .icon-shape .label{text-anchor:middle;}#mermaid-svg-G59mz7QKbU7TdMoD .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-G59mz7QKbU7TdMoD .rough-node .label,#mermaid-svg-G59mz7QKbU7TdMoD .node .label,#mermaid-svg-G59mz7QKbU7TdMoD .image-shape .label,#mermaid-svg-G59mz7QKbU7TdMoD .icon-shape .label{text-align:center;}#mermaid-svg-G59mz7QKbU7TdMoD .node.clickable{cursor:pointer;}#mermaid-svg-G59mz7QKbU7TdMoD .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-G59mz7QKbU7TdMoD .arrowheadPath{fill:#333333;}#mermaid-svg-G59mz7QKbU7TdMoD .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-G59mz7QKbU7TdMoD .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-G59mz7QKbU7TdMoD .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-G59mz7QKbU7TdMoD .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-G59mz7QKbU7TdMoD .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-G59mz7QKbU7TdMoD .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-G59mz7QKbU7TdMoD .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-G59mz7QKbU7TdMoD .cluster text{fill:#333;}#mermaid-svg-G59mz7QKbU7TdMoD .cluster span{color:#333;}#mermaid-svg-G59mz7QKbU7TdMoD div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-G59mz7QKbU7TdMoD .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-G59mz7QKbU7TdMoD rect.text{fill:none;stroke-width:0;}#mermaid-svg-G59mz7QKbU7TdMoD .icon-shape,#mermaid-svg-G59mz7QKbU7TdMoD .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-G59mz7QKbU7TdMoD .icon-shape p,#mermaid-svg-G59mz7QKbU7TdMoD .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-G59mz7QKbU7TdMoD .icon-shape rect,#mermaid-svg-G59mz7QKbU7TdMoD .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-G59mz7QKbU7TdMoD .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-G59mz7QKbU7TdMoD .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-G59mz7QKbU7TdMoD :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

自然语言描述

截图

VL 模型理解 + 定位

输出坐标/bbox

Playwright 执行操作

具象化完成

关键点：

• 不依赖 DOM，所见即所得
• 模型需正确配置 MIDSCENE_MODEL_FAMILY，否则定位坐标会偏移
• 截图清晰度、分辨率会影响定位精度

5.3 与 DOM 选择器具象性对比

六、端到端流程总览

单步操作（aiTap）流程：

#mermaid-svg-o5kBQKhpprLaLGUA{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-o5kBQKhpprLaLGUA .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-o5kBQKhpprLaLGUA .error-icon{fill:#552222;}#mermaid-svg-o5kBQKhpprLaLGUA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-o5kBQKhpprLaLGUA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-o5kBQKhpprLaLGUA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-o5kBQKhpprLaLGUA .marker.cross{stroke:#333333;}#mermaid-svg-o5kBQKhpprLaLGUA svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-o5kBQKhpprLaLGUA p{margin:0;}#mermaid-svg-o5kBQKhpprLaLGUA .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-o5kBQKhpprLaLGUA .cluster-label text{fill:#333;}#mermaid-svg-o5kBQKhpprLaLGUA .cluster-label span{color:#333;}#mermaid-svg-o5kBQKhpprLaLGUA .cluster-label span p{background-color:transparent;}#mermaid-svg-o5kBQKhpprLaLGUA .label text,#mermaid-svg-o5kBQKhpprLaLGUA span{fill:#333;color:#333;}#mermaid-svg-o5kBQKhpprLaLGUA .node rect,#mermaid-svg-o5kBQKhpprLaLGUA .node circle,#mermaid-svg-o5kBQKhpprLaLGUA .node ellipse,#mermaid-svg-o5kBQKhpprLaLGUA .node polygon,#mermaid-svg-o5kBQKhpprLaLGUA .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-o5kBQKhpprLaLGUA .rough-node .label text,#mermaid-svg-o5kBQKhpprLaLGUA .node .label text,#mermaid-svg-o5kBQKhpprLaLGUA .image-shape .label,#mermaid-svg-o5kBQKhpprLaLGUA .icon-shape .label{text-anchor:middle;}#mermaid-svg-o5kBQKhpprLaLGUA .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-o5kBQKhpprLaLGUA .rough-node .label,#mermaid-svg-o5kBQKhpprLaLGUA .node .label,#mermaid-svg-o5kBQKhpprLaLGUA .image-shape .label,#mermaid-svg-o5kBQKhpprLaLGUA .icon-shape .label{text-align:center;}#mermaid-svg-o5kBQKhpprLaLGUA .node.clickable{cursor:pointer;}#mermaid-svg-o5kBQKhpprLaLGUA .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-o5kBQKhpprLaLGUA .arrowheadPath{fill:#333333;}#mermaid-svg-o5kBQKhpprLaLGUA .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-o5kBQKhpprLaLGUA .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-o5kBQKhpprLaLGUA .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-o5kBQKhpprLaLGUA .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-o5kBQKhpprLaLGUA .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-o5kBQKhpprLaLGUA .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-o5kBQKhpprLaLGUA .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-o5kBQKhpprLaLGUA .cluster text{fill:#333;}#mermaid-svg-o5kBQKhpprLaLGUA .cluster span{color:#333;}#mermaid-svg-o5kBQKhpprLaLGUA div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-o5kBQKhpprLaLGUA .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-o5kBQKhpprLaLGUA rect.text{fill:none;stroke-width:0;}#mermaid-svg-o5kBQKhpprLaLGUA .icon-shape,#mermaid-svg-o5kBQKhpprLaLGUA .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-o5kBQKhpprLaLGUA .icon-shape p,#mermaid-svg-o5kBQKhpprLaLGUA .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-o5kBQKhpprLaLGUA .icon-shape rect,#mermaid-svg-o5kBQKhpprLaLGUA .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-o5kBQKhpprLaLGUA .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-o5kBQKhpprLaLGUA .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-o5kBQKhpprLaLGUA :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

开始

获取页面截图

截图 + 指令送入 VL 模型

模型返回元素坐标

Playwright 执行 click

结束

多步任务（aiAct）流程：

#mermaid-svg-gbt3tgcPyqKuejYr{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-gbt3tgcPyqKuejYr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-gbt3tgcPyqKuejYr .error-icon{fill:#552222;}#mermaid-svg-gbt3tgcPyqKuejYr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-gbt3tgcPyqKuejYr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-gbt3tgcPyqKuejYr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-gbt3tgcPyqKuejYr .marker.cross{stroke:#333333;}#mermaid-svg-gbt3tgcPyqKuejYr svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-gbt3tgcPyqKuejYr p{margin:0;}#mermaid-svg-gbt3tgcPyqKuejYr .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-gbt3tgcPyqKuejYr .cluster-label text{fill:#333;}#mermaid-svg-gbt3tgcPyqKuejYr .cluster-label span{color:#333;}#mermaid-svg-gbt3tgcPyqKuejYr .cluster-label span p{background-color:transparent;}#mermaid-svg-gbt3tgcPyqKuejYr .label text,#mermaid-svg-gbt3tgcPyqKuejYr span{fill:#333;color:#333;}#mermaid-svg-gbt3tgcPyqKuejYr .node rect,#mermaid-svg-gbt3tgcPyqKuejYr .node circle,#mermaid-svg-gbt3tgcPyqKuejYr .node ellipse,#mermaid-svg-gbt3tgcPyqKuejYr .node polygon,#mermaid-svg-gbt3tgcPyqKuejYr .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-gbt3tgcPyqKuejYr .rough-node .label text,#mermaid-svg-gbt3tgcPyqKuejYr .node .label text,#mermaid-svg-gbt3tgcPyqKuejYr .image-shape .label,#mermaid-svg-gbt3tgcPyqKuejYr .icon-shape .label{text-anchor:middle;}#mermaid-svg-gbt3tgcPyqKuejYr .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-gbt3tgcPyqKuejYr .rough-node .label,#mermaid-svg-gbt3tgcPyqKuejYr .node .label,#mermaid-svg-gbt3tgcPyqKuejYr .image-shape .label,#mermaid-svg-gbt3tgcPyqKuejYr .icon-shape .label{text-align:center;}#mermaid-svg-gbt3tgcPyqKuejYr .node.clickable{cursor:pointer;}#mermaid-svg-gbt3tgcPyqKuejYr .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-gbt3tgcPyqKuejYr .arrowheadPath{fill:#333333;}#mermaid-svg-gbt3tgcPyqKuejYr .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-gbt3tgcPyqKuejYr .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-gbt3tgcPyqKuejYr .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gbt3tgcPyqKuejYr .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-gbt3tgcPyqKuejYr .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gbt3tgcPyqKuejYr .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-gbt3tgcPyqKuejYr .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-gbt3tgcPyqKuejYr .cluster text{fill:#333;}#mermaid-svg-gbt3tgcPyqKuejYr .cluster span{color:#333;}#mermaid-svg-gbt3tgcPyqKuejYr div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-gbt3tgcPyqKuejYr .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-gbt3tgcPyqKuejYr rect.text{fill:none;stroke-width:0;}#mermaid-svg-gbt3tgcPyqKuejYr .icon-shape,#mermaid-svg-gbt3tgcPyqKuejYr .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gbt3tgcPyqKuejYr .icon-shape p,#mermaid-svg-gbt3tgcPyqKuejYr .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-gbt3tgcPyqKuejYr .icon-shape rect,#mermaid-svg-gbt3tgcPyqKuejYr .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gbt3tgcPyqKuejYr .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-gbt3tgcPyqKuejYr .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-gbt3tgcPyqKuejYr :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

是

否

开始

Planning 模型规划步骤列表

取第一步

视觉定位该步目标

执行操作

还有后续步骤?

结束

七、简单案例：百度搜索

下面是一个完整的可运行示例，演示用 Midscene + Playwright 在百度搜索关键词并提取第一条结果标题。

7.1 项目准备

mkdir midscene-demo && cd midscene-demo
npm init -y
npm install @midscene/web playwright tsx dotenv –save-dev

创建 .env 文件配置模型（以千问为例）：

MIDSCENE_MODEL_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
MIDSCENE_MODEL_API_KEY=你的API密钥
MIDSCENE_MODEL_NAME=qwen3-vl-plus
MIDSCENE_MODEL_FAMILY=qwen3-vl

7.2 完整代码

创建 search-baidu.ts：

import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';
import 'dotenv/config';

async function main() {
const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
await page.setViewportSize({ width: 1280, height: 768 });

// 打开百度
await page.goto('https://www.baidu.com');
await page.waitForLoadState('networkidle');

const agent = new PlaywrightAgent(page);

// 在搜索框输入关键词并搜索
await agent.aiInput('Playwright 自动化', '搜索框');
await agent.aiTap('百度一下按钮');

// 等待搜索结果加载
await agent.aiWaitFor('搜索结果列表已出现', { timeoutMs: 5000 });

// 提取第一条结果的标题
const firstTitle = await agent.aiString('第一条搜索结果的标题是什么？');
console.log('第一条结果标题:', firstTitle);

await browser.close();
}

main().catch(console.error);

7.3 运行

npx tsx search-baidu.ts

7.4 案例流程图

#mermaid-svg-anvNnOZc2kdQjmdB{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-anvNnOZc2kdQjmdB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-anvNnOZc2kdQjmdB .error-icon{fill:#552222;}#mermaid-svg-anvNnOZc2kdQjmdB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-anvNnOZc2kdQjmdB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-anvNnOZc2kdQjmdB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-anvNnOZc2kdQjmdB .marker.cross{stroke:#333333;}#mermaid-svg-anvNnOZc2kdQjmdB svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-anvNnOZc2kdQjmdB p{margin:0;}#mermaid-svg-anvNnOZc2kdQjmdB .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-anvNnOZc2kdQjmdB .cluster-label text{fill:#333;}#mermaid-svg-anvNnOZc2kdQjmdB .cluster-label span{color:#333;}#mermaid-svg-anvNnOZc2kdQjmdB .cluster-label span p{background-color:transparent;}#mermaid-svg-anvNnOZc2kdQjmdB .label text,#mermaid-svg-anvNnOZc2kdQjmdB span{fill:#333;color:#333;}#mermaid-svg-anvNnOZc2kdQjmdB .node rect,#mermaid-svg-anvNnOZc2kdQjmdB .node circle,#mermaid-svg-anvNnOZc2kdQjmdB .node ellipse,#mermaid-svg-anvNnOZc2kdQjmdB .node polygon,#mermaid-svg-anvNnOZc2kdQjmdB .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-anvNnOZc2kdQjmdB .rough-node .label text,#mermaid-svg-anvNnOZc2kdQjmdB .node .label text,#mermaid-svg-anvNnOZc2kdQjmdB .image-shape .label,#mermaid-svg-anvNnOZc2kdQjmdB .icon-shape .label{text-anchor:middle;}#mermaid-svg-anvNnOZc2kdQjmdB .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-anvNnOZc2kdQjmdB .rough-node .label,#mermaid-svg-anvNnOZc2kdQjmdB .node .label,#mermaid-svg-anvNnOZc2kdQjmdB .image-shape .label,#mermaid-svg-anvNnOZc2kdQjmdB .icon-shape .label{text-align:center;}#mermaid-svg-anvNnOZc2kdQjmdB .node.clickable{cursor:pointer;}#mermaid-svg-anvNnOZc2kdQjmdB .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-anvNnOZc2kdQjmdB .arrowheadPath{fill:#333333;}#mermaid-svg-anvNnOZc2kdQjmdB .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-anvNnOZc2kdQjmdB .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-anvNnOZc2kdQjmdB .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-anvNnOZc2kdQjmdB .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-anvNnOZc2kdQjmdB .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-anvNnOZc2kdQjmdB .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-anvNnOZc2kdQjmdB .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-anvNnOZc2kdQjmdB .cluster text{fill:#333;}#mermaid-svg-anvNnOZc2kdQjmdB .cluster span{color:#333;}#mermaid-svg-anvNnOZc2kdQjmdB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-anvNnOZc2kdQjmdB .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-anvNnOZc2kdQjmdB rect.text{fill:none;stroke-width:0;}#mermaid-svg-anvNnOZc2kdQjmdB .icon-shape,#mermaid-svg-anvNnOZc2kdQjmdB .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-anvNnOZc2kdQjmdB .icon-shape p,#mermaid-svg-anvNnOZc2kdQjmdB .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-anvNnOZc2kdQjmdB .icon-shape rect,#mermaid-svg-anvNnOZc2kdQjmdB .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-anvNnOZc2kdQjmdB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-anvNnOZc2kdQjmdB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-anvNnOZc2kdQjmdB :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

启动

打开百度首页

aiInput: 在搜索框输入关键词

aiTap: 点击百度一下

aiWaitFor: 等待结果加载

aiString: 提取第一条标题

打印结果

结束

八、Playwright 与 Midscene 对比总结

九、最佳实践与注意事项

8.1 最佳实践

8.2 当前局限

• 依赖视觉模型质量，定位精度受模型能力影响
• 需正确配置 MIDSCENE_MODEL_FAMILY，否则坐标偏移
• 复杂任务可配合 Planning/Insight 多模型提升成功率

十、总结

Midscene 让 UI 自动化从「写选择器」走向「说人话」，是 AI 时代浏览器与跨平台自动化的有力工具。

附录：完整 Demo 项目

下面是一个可直接运行的 Demo 项目结构，包含两种写法：即时操作（aiTap/aiInput）和 自动规划（aiAct）。

项目结构

midscene-demo/
├── package.json
├── .env
└── demo.ts

package.json

{
"name": "midscene-demo",
"type": "module",
"scripts": {
"demo": "tsx demo.ts"
},
"devDependencies": {
"@midscene/web": "latest",
"dotenv": "^16.0.0",
"playwright": "latest",
"tsx": "^4.0.0"
}
}

.env（复制后填入你的 API Key）

MIDSCENE_MODEL_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
MIDSCENE_MODEL_API_KEY=sk-xxxxxxxx
MIDSCENE_MODEL_NAME=qwen3-vl-plus
MIDSCENE_MODEL_FAMILY=qwen3-vl

demo.ts（完整代码）

import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';
import 'dotenv/config';

async function runDemo() {
const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
await page.setViewportSize({ width: 1280, height: 768 });

await page.goto('https://www.baidu.com');
await page.waitForLoadState('networkidle');

const agent = new PlaywrightAgent(page);

// ========== 写法一：即时操作（推荐，步骤清晰） ==========
await agent.aiInput('Midscene 自动化', '搜索框');
await agent.aiTap('百度一下');
await agent.aiWaitFor('搜索结果已加载', { timeoutMs: 8000 });

const title = await agent.aiString('第一条搜索结果的标题');
console.log('【即时操作】第一条结果:', title);

// ========== 写法二：自动规划（一句话完成多步） ==========
// 先回到首页
await page.goto('https://www.baidu.com');
await page.waitForLoadState('networkidle');

await agent.aiAct('在搜索框输入"Playwright"，点击百度一下，等待结果出现');
const title2 = await agent.aiString('第一条搜索结果的标题');
console.log('【自动规划】第一条结果:', title2);

await browser.close();
}

runDemo().catch(console.error);

运行步骤

# 1. 创建项目
mkdir midscene-demo && cd midscene-demo

# 2. 安装依赖
npm install @midscene/web playwright tsx dotenv –save-dev

# 3. 首次需安装浏览器
npx playwright install chromium

# 4. 配置 .env 后运行
npm run demo

代码要点说明