电商商品全网聚合平台 - 技术实现方案

搜索 + 比价 | 多平台数据聚合 | 混合检索

一、系统架构总览

#mermaid-svg-h2652FpwyP0a8N0g{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-h2652FpwyP0a8N0g .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-h2652FpwyP0a8N0g .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-h2652FpwyP0a8N0g .error-icon{fill:#552222;}#mermaid-svg-h2652FpwyP0a8N0g .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-h2652FpwyP0a8N0g .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-h2652FpwyP0a8N0g .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-h2652FpwyP0a8N0g .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-h2652FpwyP0a8N0g .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-h2652FpwyP0a8N0g .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-h2652FpwyP0a8N0g .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-h2652FpwyP0a8N0g .marker{fill:#333333;stroke:#333333;}#mermaid-svg-h2652FpwyP0a8N0g .marker.cross{stroke:#333333;}#mermaid-svg-h2652FpwyP0a8N0g svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-h2652FpwyP0a8N0g p{margin:0;}#mermaid-svg-h2652FpwyP0a8N0g .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-h2652FpwyP0a8N0g .cluster-label text{fill:#333;}#mermaid-svg-h2652FpwyP0a8N0g .cluster-label span{color:#333;}#mermaid-svg-h2652FpwyP0a8N0g .cluster-label span p{background-color:transparent;}#mermaid-svg-h2652FpwyP0a8N0g .label text,#mermaid-svg-h2652FpwyP0a8N0g span{fill:#333;color:#333;}#mermaid-svg-h2652FpwyP0a8N0g .node rect,#mermaid-svg-h2652FpwyP0a8N0g .node circle,#mermaid-svg-h2652FpwyP0a8N0g .node ellipse,#mermaid-svg-h2652FpwyP0a8N0g .node polygon,#mermaid-svg-h2652FpwyP0a8N0g .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-h2652FpwyP0a8N0g .rough-node .label text,#mermaid-svg-h2652FpwyP0a8N0g .node .label text,#mermaid-svg-h2652FpwyP0a8N0g .image-shape .label,#mermaid-svg-h2652FpwyP0a8N0g .icon-shape .label{text-anchor:middle;}#mermaid-svg-h2652FpwyP0a8N0g .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-h2652FpwyP0a8N0g .rough-node .label,#mermaid-svg-h2652FpwyP0a8N0g .node .label,#mermaid-svg-h2652FpwyP0a8N0g .image-shape .label,#mermaid-svg-h2652FpwyP0a8N0g .icon-shape .label{text-align:center;}#mermaid-svg-h2652FpwyP0a8N0g .node.clickable{cursor:pointer;}#mermaid-svg-h2652FpwyP0a8N0g .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-h2652FpwyP0a8N0g .arrowheadPath{fill:#333333;}#mermaid-svg-h2652FpwyP0a8N0g .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-h2652FpwyP0a8N0g .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-h2652FpwyP0a8N0g .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-h2652FpwyP0a8N0g .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-h2652FpwyP0a8N0g .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-h2652FpwyP0a8N0g .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-h2652FpwyP0a8N0g .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-h2652FpwyP0a8N0g .cluster text{fill:#333;}#mermaid-svg-h2652FpwyP0a8N0g .cluster span{color:#333;}#mermaid-svg-h2652FpwyP0a8N0g div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-h2652FpwyP0a8N0g .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-h2652FpwyP0a8N0g rect.text{fill:none;stroke-width:0;}#mermaid-svg-h2652FpwyP0a8N0g .icon-shape,#mermaid-svg-h2652FpwyP0a8N0g .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-h2652FpwyP0a8N0g .icon-shape p,#mermaid-svg-h2652FpwyP0a8N0g .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-h2652FpwyP0a8N0g .icon-shape .label rect,#mermaid-svg-h2652FpwyP0a8N0g .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-h2652FpwyP0a8N0g .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-h2652FpwyP0a8N0g .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-h2652FpwyP0a8N0g :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

数据存储层

Elasticsearch搜索+向量索引

MySQL业务数据/元数据

Redis缓存/分布式锁/限流

数据处理层

数据采集Scheduler

数据清洗ETL管道

向量化Embedding

索引构建ES写入

服务层

搜索服务

比价服务

商品服务

前端应用层

Web搜索界面 / API接口 / 数据可视化面板

二、技术栈选型

三、数据采集模块

3.1 数据源分类

3.2 采集架构设计

#mermaid-svg-rAv7BrurnQqTMwzl{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-rAv7BrurnQqTMwzl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-rAv7BrurnQqTMwzl .error-icon{fill:#552222;}#mermaid-svg-rAv7BrurnQqTMwzl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-rAv7BrurnQqTMwzl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-rAv7BrurnQqTMwzl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-rAv7BrurnQqTMwzl .marker.cross{stroke:#333333;}#mermaid-svg-rAv7BrurnQqTMwzl svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-rAv7BrurnQqTMwzl p{margin:0;}#mermaid-svg-rAv7BrurnQqTMwzl .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-rAv7BrurnQqTMwzl .cluster-label text{fill:#333;}#mermaid-svg-rAv7BrurnQqTMwzl .cluster-label span{color:#333;}#mermaid-svg-rAv7BrurnQqTMwzl .cluster-label span p{background-color:transparent;}#mermaid-svg-rAv7BrurnQqTMwzl .label text,#mermaid-svg-rAv7BrurnQqTMwzl span{fill:#333;color:#333;}#mermaid-svg-rAv7BrurnQqTMwzl .node rect,#mermaid-svg-rAv7BrurnQqTMwzl .node circle,#mermaid-svg-rAv7BrurnQqTMwzl .node ellipse,#mermaid-svg-rAv7BrurnQqTMwzl .node polygon,#mermaid-svg-rAv7BrurnQqTMwzl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-rAv7BrurnQqTMwzl .rough-node .label text,#mermaid-svg-rAv7BrurnQqTMwzl .node .label text,#mermaid-svg-rAv7BrurnQqTMwzl .image-shape .label,#mermaid-svg-rAv7BrurnQqTMwzl .icon-shape .label{text-anchor:middle;}#mermaid-svg-rAv7BrurnQqTMwzl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-rAv7BrurnQqTMwzl .rough-node .label,#mermaid-svg-rAv7BrurnQqTMwzl .node .label,#mermaid-svg-rAv7BrurnQqTMwzl .image-shape .label,#mermaid-svg-rAv7BrurnQqTMwzl .icon-shape .label{text-align:center;}#mermaid-svg-rAv7BrurnQqTMwzl .node.clickable{cursor:pointer;}#mermaid-svg-rAv7BrurnQqTMwzl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-rAv7BrurnQqTMwzl .arrowheadPath{fill:#333333;}#mermaid-svg-rAv7BrurnQqTMwzl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-rAv7BrurnQqTMwzl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-rAv7BrurnQqTMwzl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-rAv7BrurnQqTMwzl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-rAv7BrurnQqTMwzl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-rAv7BrurnQqTMwzl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-rAv7BrurnQqTMwzl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-rAv7BrurnQqTMwzl .cluster text{fill:#333;}#mermaid-svg-rAv7BrurnQqTMwzl .cluster span{color:#333;}#mermaid-svg-rAv7BrurnQqTMwzl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-rAv7BrurnQqTMwzl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-rAv7BrurnQqTMwzl rect.text{fill:none;stroke-width:0;}#mermaid-svg-rAv7BrurnQqTMwzl .icon-shape,#mermaid-svg-rAv7BrurnQqTMwzl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-rAv7BrurnQqTMwzl .icon-shape p,#mermaid-svg-rAv7BrurnQqTMwzl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-rAv7BrurnQqTMwzl .icon-shape .label rect,#mermaid-svg-rAv7BrurnQqTMwzl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-rAv7BrurnQqTMwzl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-rAv7BrurnQqTMwzl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-rAv7BrurnQqTMwzl :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

采集调度中心 (XXL-JOB)

京东采集 Worker

拼多多 Worker

1688采集 Worker

抖音采集 Worker

同步处理管道

1. 数据采集

2. 数据清洗

3. 向量化

4. 写入 ES

5. 更新 MySQL

3.3 各平台API接入说明

3.3.1 京东开放平台

接入要点：

• 请求参数按字母排序后拼接签名
• 时间戳使用毫秒级
• 需处理API限流，实现重试机制
• 商品数据需定时增量更新

3.3.2 拼多多开放平台

接入要点：

• 时间戳使用秒级
• POST请求，参数放在body中
• 多多进宝接口可获取推广佣金信息
• 部分接口需要用户授权token

3.3.3 1688开放平台

接入要点：

• URL路径包含appKey和接口名称
• 批发商品有起订量、阶梯价格等特殊字段
• 部分数据需要企业账号权限

3.3.4 抖音电商开放平台

接入要点：

• AccessToken需缓存，提前5分钟刷新
• 部分接口需要用户授权
• 直播商品数据实时更新频率高

3.4 自研爬虫模块

技术选型： Selenium WebDriver (Java)

爬虫架构：

#mermaid-svg-dle9eDZz06FwIuv3{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dle9eDZz06FwIuv3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dle9eDZz06FwIuv3 .error-icon{fill:#552222;}#mermaid-svg-dle9eDZz06FwIuv3 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dle9eDZz06FwIuv3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dle9eDZz06FwIuv3 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dle9eDZz06FwIuv3 .marker.cross{stroke:#333333;}#mermaid-svg-dle9eDZz06FwIuv3 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dle9eDZz06FwIuv3 p{margin:0;}#mermaid-svg-dle9eDZz06FwIuv3 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-dle9eDZz06FwIuv3 .cluster-label text{fill:#333;}#mermaid-svg-dle9eDZz06FwIuv3 .cluster-label span{color:#333;}#mermaid-svg-dle9eDZz06FwIuv3 .cluster-label span p{background-color:transparent;}#mermaid-svg-dle9eDZz06FwIuv3 .label text,#mermaid-svg-dle9eDZz06FwIuv3 span{fill:#333;color:#333;}#mermaid-svg-dle9eDZz06FwIuv3 .node rect,#mermaid-svg-dle9eDZz06FwIuv3 .node circle,#mermaid-svg-dle9eDZz06FwIuv3 .node ellipse,#mermaid-svg-dle9eDZz06FwIuv3 .node polygon,#mermaid-svg-dle9eDZz06FwIuv3 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dle9eDZz06FwIuv3 .rough-node .label text,#mermaid-svg-dle9eDZz06FwIuv3 .node .label text,#mermaid-svg-dle9eDZz06FwIuv3 .image-shape .label,#mermaid-svg-dle9eDZz06FwIuv3 .icon-shape .label{text-anchor:middle;}#mermaid-svg-dle9eDZz06FwIuv3 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dle9eDZz06FwIuv3 .rough-node .label,#mermaid-svg-dle9eDZz06FwIuv3 .node .label,#mermaid-svg-dle9eDZz06FwIuv3 .image-shape .label,#mermaid-svg-dle9eDZz06FwIuv3 .icon-shape .label{text-align:center;}#mermaid-svg-dle9eDZz06FwIuv3 .node.clickable{cursor:pointer;}#mermaid-svg-dle9eDZz06FwIuv3 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dle9eDZz06FwIuv3 .arrowheadPath{fill:#333333;}#mermaid-svg-dle9eDZz06FwIuv3 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dle9eDZz06FwIuv3 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dle9eDZz06FwIuv3 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dle9eDZz06FwIuv3 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dle9eDZz06FwIuv3 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dle9eDZz06FwIuv3 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dle9eDZz06FwIuv3 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dle9eDZz06FwIuv3 .cluster text{fill:#333;}#mermaid-svg-dle9eDZz06FwIuv3 .cluster span{color:#333;}#mermaid-svg-dle9eDZz06FwIuv3 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dle9eDZz06FwIuv3 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dle9eDZz06FwIuv3 rect.text{fill:none;stroke-width:0;}#mermaid-svg-dle9eDZz06FwIuv3 .icon-shape,#mermaid-svg-dle9eDZz06FwIuv3 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dle9eDZz06FwIuv3 .icon-shape p,#mermaid-svg-dle9eDZz06FwIuv3 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dle9eDZz06FwIuv3 .icon-shape .label rect,#mermaid-svg-dle9eDZz06FwIuv3 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dle9eDZz06FwIuv3 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dle9eDZz06FwIuv3 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dle9eDZz06FwIuv3 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

爬虫调度器

BaseCrawler 抽象基类createDriver() 创建浏览器驱动crawl() 执行爬取流程search() 搜索商品(抽象)parseProduct() 解析商品(抽象)

TBCrawler(淘宝/天猫)

其他爬虫(扩展)

反爬策略：

• 无头浏览器模式运行
• 移除 webdriver 特征标识
• 设置合理的请求间隔（2-5秒）
• 使用代理IP池轮换
• Cookie池管理

数据采集流程：

四、数据清洗与归一化模块

4.1 数据清洗流程总览

#mermaid-svg-lsX8MIh14qSbwx7Z{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-lsX8MIh14qSbwx7Z .error-icon{fill:#552222;}#mermaid-svg-lsX8MIh14qSbwx7Z .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-lsX8MIh14qSbwx7Z .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-lsX8MIh14qSbwx7Z .marker{fill:#333333;stroke:#333333;}#mermaid-svg-lsX8MIh14qSbwx7Z .marker.cross{stroke:#333333;}#mermaid-svg-lsX8MIh14qSbwx7Z svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-lsX8MIh14qSbwx7Z p{margin:0;}#mermaid-svg-lsX8MIh14qSbwx7Z .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z .cluster-label text{fill:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z .cluster-label span{color:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z .cluster-label span p{background-color:transparent;}#mermaid-svg-lsX8MIh14qSbwx7Z .label text,#mermaid-svg-lsX8MIh14qSbwx7Z span{fill:#333;color:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z .node rect,#mermaid-svg-lsX8MIh14qSbwx7Z .node circle,#mermaid-svg-lsX8MIh14qSbwx7Z .node ellipse,#mermaid-svg-lsX8MIh14qSbwx7Z .node polygon,#mermaid-svg-lsX8MIh14qSbwx7Z .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-lsX8MIh14qSbwx7Z .rough-node .label text,#mermaid-svg-lsX8MIh14qSbwx7Z .node .label text,#mermaid-svg-lsX8MIh14qSbwx7Z .image-shape .label,#mermaid-svg-lsX8MIh14qSbwx7Z .icon-shape .label{text-anchor:middle;}#mermaid-svg-lsX8MIh14qSbwx7Z .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-lsX8MIh14qSbwx7Z .rough-node .label,#mermaid-svg-lsX8MIh14qSbwx7Z .node .label,#mermaid-svg-lsX8MIh14qSbwx7Z .image-shape .label,#mermaid-svg-lsX8MIh14qSbwx7Z .icon-shape .label{text-align:center;}#mermaid-svg-lsX8MIh14qSbwx7Z .node.clickable{cursor:pointer;}#mermaid-svg-lsX8MIh14qSbwx7Z .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-lsX8MIh14qSbwx7Z .arrowheadPath{fill:#333333;}#mermaid-svg-lsX8MIh14qSbwx7Z .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-lsX8MIh14qSbwx7Z .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-lsX8MIh14qSbwx7Z .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lsX8MIh14qSbwx7Z .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-lsX8MIh14qSbwx7Z .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lsX8MIh14qSbwx7Z .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-lsX8MIh14qSbwx7Z .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-lsX8MIh14qSbwx7Z .cluster text{fill:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z .cluster span{color:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-lsX8MIh14qSbwx7Z .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-lsX8MIh14qSbwx7Z rect.text{fill:none;stroke-width:0;}#mermaid-svg-lsX8MIh14qSbwx7Z .icon-shape,#mermaid-svg-lsX8MIh14qSbwx7Z .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lsX8MIh14qSbwx7Z .icon-shape p,#mermaid-svg-lsX8MIh14qSbwx7Z .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-lsX8MIh14qSbwx7Z .icon-shape .label rect,#mermaid-svg-lsX8MIh14qSbwx7Z .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lsX8MIh14qSbwx7Z .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-lsX8MIh14qSbwx7Z .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-lsX8MIh14qSbwx7Z :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

原始数据(Raw JSON)

格式校验 & 过滤结构校验 / 空值检查异常拦截 / 类型转换

字段提取 & 转换平台映射 / 字段转换

数据标准化 & 归一化价格/文本/图片时间/类目归一化

去重过滤 & 合并唯一标识 / 模糊匹配版本控制

质量评分 & 入库质量打分 / 分级入库异常标记

多平台原始JSON

校验规则

字段映射表

标准化规则

去重算法

质量等级 A/B/C/D

4.2 详细清洗步骤

4.2.1 第一步：格式校验与过滤

目标： 拦截无效数据，确保进入后续流程的数据结构正确

异常处理策略：

• 轻微异常（可修复）：尝试自动修复后继续流程
• 严重异常（不可修复）：丢弃数据，写入异常日志表
• 可疑异常（需人工判断）：写入待审核队列

4.2.2 第二步：字段提取与转换

目标： 将各平台异构数据映射为统一数据模型

平台字段映射表：

类型转换规则：

4.2.3 第三步：数据标准化与归一化

（1）价格标准化

（2）文本清洗

广告词黑名单：

（3）图片URL标准化

（4）类目归一化

（5）时间标准化

4.2.4 第四步：去重与合并

（1）精确去重

（2）模糊去重算法

相似度计算 = 0.40 × 标题语义相似度
+ 0.25 × 品牌匹配度
+ 0.20 × 类目匹配度
+ 0.15 × 价格接近度

判定规则：
– 相似度 >= 0.85：视为同款商品，合并数据
– 0.70 <= 相似度 < 0.85：疑似同款，进入人工审核
– 相似度 < 0.70：视为不同商品

（3）数据合并策略

当识别为同款商品时：

（4）版本控制

4.2.5 第五步：质量评分与分级

质量评分维度：

质量分级：

4.3 统一商品数据模型

4.4 异常数据处理

异常类型与处理：

异常数据流转：

#mermaid-svg-F8gSjGIoSqF40OrM{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-F8gSjGIoSqF40OrM .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-F8gSjGIoSqF40OrM .error-icon{fill:#552222;}#mermaid-svg-F8gSjGIoSqF40OrM .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-F8gSjGIoSqF40OrM .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-F8gSjGIoSqF40OrM .marker{fill:#333333;stroke:#333333;}#mermaid-svg-F8gSjGIoSqF40OrM .marker.cross{stroke:#333333;}#mermaid-svg-F8gSjGIoSqF40OrM svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-F8gSjGIoSqF40OrM p{margin:0;}#mermaid-svg-F8gSjGIoSqF40OrM .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-F8gSjGIoSqF40OrM .cluster-label text{fill:#333;}#mermaid-svg-F8gSjGIoSqF40OrM .cluster-label span{color:#333;}#mermaid-svg-F8gSjGIoSqF40OrM .cluster-label span p{background-color:transparent;}#mermaid-svg-F8gSjGIoSqF40OrM .label text,#mermaid-svg-F8gSjGIoSqF40OrM span{fill:#333;color:#333;}#mermaid-svg-F8gSjGIoSqF40OrM .node rect,#mermaid-svg-F8gSjGIoSqF40OrM .node circle,#mermaid-svg-F8gSjGIoSqF40OrM .node ellipse,#mermaid-svg-F8gSjGIoSqF40OrM .node polygon,#mermaid-svg-F8gSjGIoSqF40OrM .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-F8gSjGIoSqF40OrM .rough-node .label text,#mermaid-svg-F8gSjGIoSqF40OrM .node .label text,#mermaid-svg-F8gSjGIoSqF40OrM .image-shape .label,#mermaid-svg-F8gSjGIoSqF40OrM .icon-shape .label{text-anchor:middle;}#mermaid-svg-F8gSjGIoSqF40OrM .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-F8gSjGIoSqF40OrM .rough-node .label,#mermaid-svg-F8gSjGIoSqF40OrM .node .label,#mermaid-svg-F8gSjGIoSqF40OrM .image-shape .label,#mermaid-svg-F8gSjGIoSqF40OrM .icon-shape .label{text-align:center;}#mermaid-svg-F8gSjGIoSqF40OrM .node.clickable{cursor:pointer;}#mermaid-svg-F8gSjGIoSqF40OrM .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-F8gSjGIoSqF40OrM .arrowheadPath{fill:#333333;}#mermaid-svg-F8gSjGIoSqF40OrM .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-F8gSjGIoSqF40OrM .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-F8gSjGIoSqF40OrM .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-F8gSjGIoSqF40OrM .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-F8gSjGIoSqF40OrM .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-F8gSjGIoSqF40OrM .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-F8gSjGIoSqF40OrM .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-F8gSjGIoSqF40OrM .cluster text{fill:#333;}#mermaid-svg-F8gSjGIoSqF40OrM .cluster span{color:#333;}#mermaid-svg-F8gSjGIoSqF40OrM div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-F8gSjGIoSqF40OrM .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-F8gSjGIoSqF40OrM rect.text{fill:none;stroke-width:0;}#mermaid-svg-F8gSjGIoSqF40OrM .icon-shape,#mermaid-svg-F8gSjGIoSqF40OrM .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-F8gSjGIoSqF40OrM .icon-shape p,#mermaid-svg-F8gSjGIoSqF40OrM .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-F8gSjGIoSqF40OrM .icon-shape .label rect,#mermaid-svg-F8gSjGIoSqF40OrM .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-F8gSjGIoSqF40OrM .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-F8gSjGIoSqF40OrM .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-F8gSjGIoSqF40OrM :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

是

否

异常数据

异常分类

自动修复

修复成功?

正常流程

修复失败

人工审核

4.5 清洗流程性能优化

五、数据存储与索引构建

5.1 Elasticsearch 索引设计

5.1.1 索引配置

5.1.2 字段映射

5.2 向量化服务

向量模型选型：

向量化流程：

商品入库时，对同一份商品数据同步执行向量化：

#mermaid-svg-HDEMOtho4VEVW5cF{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-HDEMOtho4VEVW5cF .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-HDEMOtho4VEVW5cF .error-icon{fill:#552222;}#mermaid-svg-HDEMOtho4VEVW5cF .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-HDEMOtho4VEVW5cF .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-HDEMOtho4VEVW5cF .marker{fill:#333333;stroke:#333333;}#mermaid-svg-HDEMOtho4VEVW5cF .marker.cross{stroke:#333333;}#mermaid-svg-HDEMOtho4VEVW5cF svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-HDEMOtho4VEVW5cF p{margin:0;}#mermaid-svg-HDEMOtho4VEVW5cF .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-HDEMOtho4VEVW5cF .cluster-label text{fill:#333;}#mermaid-svg-HDEMOtho4VEVW5cF .cluster-label span{color:#333;}#mermaid-svg-HDEMOtho4VEVW5cF .cluster-label span p{background-color:transparent;}#mermaid-svg-HDEMOtho4VEVW5cF .label text,#mermaid-svg-HDEMOtho4VEVW5cF span{fill:#333;color:#333;}#mermaid-svg-HDEMOtho4VEVW5cF .node rect,#mermaid-svg-HDEMOtho4VEVW5cF .node circle,#mermaid-svg-HDEMOtho4VEVW5cF .node ellipse,#mermaid-svg-HDEMOtho4VEVW5cF .node polygon,#mermaid-svg-HDEMOtho4VEVW5cF .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-HDEMOtho4VEVW5cF .rough-node .label text,#mermaid-svg-HDEMOtho4VEVW5cF .node .label text,#mermaid-svg-HDEMOtho4VEVW5cF .image-shape .label,#mermaid-svg-HDEMOtho4VEVW5cF .icon-shape .label{text-anchor:middle;}#mermaid-svg-HDEMOtho4VEVW5cF .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-HDEMOtho4VEVW5cF .rough-node .label,#mermaid-svg-HDEMOtho4VEVW5cF .node .label,#mermaid-svg-HDEMOtho4VEVW5cF .image-shape .label,#mermaid-svg-HDEMOtho4VEVW5cF .icon-shape .label{text-align:center;}#mermaid-svg-HDEMOtho4VEVW5cF .node.clickable{cursor:pointer;}#mermaid-svg-HDEMOtho4VEVW5cF .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-HDEMOtho4VEVW5cF .arrowheadPath{fill:#333333;}#mermaid-svg-HDEMOtho4VEVW5cF .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-HDEMOtho4VEVW5cF .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-HDEMOtho4VEVW5cF .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-HDEMOtho4VEVW5cF .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-HDEMOtho4VEVW5cF .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-HDEMOtho4VEVW5cF .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-HDEMOtho4VEVW5cF .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-HDEMOtho4VEVW5cF .cluster text{fill:#333;}#mermaid-svg-HDEMOtho4VEVW5cF .cluster span{color:#333;}#mermaid-svg-HDEMOtho4VEVW5cF div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-HDEMOtho4VEVW5cF .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-HDEMOtho4VEVW5cF rect.text{fill:none;stroke-width:0;}#mermaid-svg-HDEMOtho4VEVW5cF .icon-shape,#mermaid-svg-HDEMOtho4VEVW5cF .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-HDEMOtho4VEVW5cF .icon-shape p,#mermaid-svg-HDEMOtho4VEVW5cF .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-HDEMOtho4VEVW5cF .icon-shape .label rect,#mermaid-svg-HDEMOtho4VEVW5cF .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-HDEMOtho4VEVW5cF .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-HDEMOtho4VEVW5cF .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-HDEMOtho4VEVW5cF :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

文本预处理

【输入】商品数据标题 / 描述 / 品牌 / 类目

标题清洗后文本标题+品牌+类目

原始标题文本

描述清洗后文本

【Qwen】标题稠密向量1024维

【SPLADE】标题稀疏向量

【Qwen】描述稠密向量1024维

写入 EStitle.dense_vector

写入 EStitle.sparse_vector

写入 ESdesc.dense_vector

向量化完成

关键说明：

• 标题向量化：Qwen（清洗后标题+品牌+类目）+ SPLADE（原始标题文本）
• 描述向量化：Qwen（清洗后描述文本），仅生成稠密向量
• 所有向量化同步执行，互不依赖
• 最终同时写入ES对应字段

批量处理策略：

• 批量向量化（每批100条）
• 同步处理，采集完直接清洗入库
• 失败重试机制
• GPU加速（可选）

5.3 索引写入策略

写入流程：

性能优化：

• 使用ES bulk API批量写入
• 每批100-500条记录
• 写入时关闭refresh_interval，完成后恢复
• 冷热数据分离（近期数据SSD，历史数据HDD）

六、数据检索服务

6.1 混合检索架构

#mermaid-svg-BOsuy2snRx3e0WyW{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-BOsuy2snRx3e0WyW .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-BOsuy2snRx3e0WyW .error-icon{fill:#552222;}#mermaid-svg-BOsuy2snRx3e0WyW .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-BOsuy2snRx3e0WyW .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-BOsuy2snRx3e0WyW .marker{fill:#333333;stroke:#333333;}#mermaid-svg-BOsuy2snRx3e0WyW .marker.cross{stroke:#333333;}#mermaid-svg-BOsuy2snRx3e0WyW svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-BOsuy2snRx3e0WyW p{margin:0;}#mermaid-svg-BOsuy2snRx3e0WyW .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-BOsuy2snRx3e0WyW .cluster-label text{fill:#333;}#mermaid-svg-BOsuy2snRx3e0WyW .cluster-label span{color:#333;}#mermaid-svg-BOsuy2snRx3e0WyW .cluster-label span p{background-color:transparent;}#mermaid-svg-BOsuy2snRx3e0WyW .label text,#mermaid-svg-BOsuy2snRx3e0WyW span{fill:#333;color:#333;}#mermaid-svg-BOsuy2snRx3e0WyW .node rect,#mermaid-svg-BOsuy2snRx3e0WyW .node circle,#mermaid-svg-BOsuy2snRx3e0WyW .node ellipse,#mermaid-svg-BOsuy2snRx3e0WyW .node polygon,#mermaid-svg-BOsuy2snRx3e0WyW .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-BOsuy2snRx3e0WyW .rough-node .label text,#mermaid-svg-BOsuy2snRx3e0WyW .node .label text,#mermaid-svg-BOsuy2snRx3e0WyW .image-shape .label,#mermaid-svg-BOsuy2snRx3e0WyW .icon-shape .label{text-anchor:middle;}#mermaid-svg-BOsuy2snRx3e0WyW .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-BOsuy2snRx3e0WyW .rough-node .label,#mermaid-svg-BOsuy2snRx3e0WyW .node .label,#mermaid-svg-BOsuy2snRx3e0WyW .image-shape .label,#mermaid-svg-BOsuy2snRx3e0WyW .icon-shape .label{text-align:center;}#mermaid-svg-BOsuy2snRx3e0WyW .node.clickable{cursor:pointer;}#mermaid-svg-BOsuy2snRx3e0WyW .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-BOsuy2snRx3e0WyW .arrowheadPath{fill:#333333;}#mermaid-svg-BOsuy2snRx3e0WyW .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-BOsuy2snRx3e0WyW .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-BOsuy2snRx3e0WyW .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-BOsuy2snRx3e0WyW .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-BOsuy2snRx3e0WyW .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-BOsuy2snRx3e0WyW .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-BOsuy2snRx3e0WyW .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-BOsuy2snRx3e0WyW .cluster text{fill:#333;}#mermaid-svg-BOsuy2snRx3e0WyW .cluster span{color:#333;}#mermaid-svg-BOsuy2snRx3e0WyW div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-BOsuy2snRx3e0WyW .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-BOsuy2snRx3e0WyW rect.text{fill:none;stroke-width:0;}#mermaid-svg-BOsuy2snRx3e0WyW .icon-shape,#mermaid-svg-BOsuy2snRx3e0WyW .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-BOsuy2snRx3e0WyW .icon-shape p,#mermaid-svg-BOsuy2snRx3e0WyW .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-BOsuy2snRx3e0WyW .icon-shape .label rect,#mermaid-svg-BOsuy2snRx3e0WyW .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-BOsuy2snRx3e0WyW .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-BOsuy2snRx3e0WyW .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-BOsuy2snRx3e0WyW :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

用户查询: iPhone 15 手机壳

关键字检索 (BM25)

稠密向量语义检索(Qwen 1024维)

稀疏向量词元检索(SPLADE)

RRF 融合排序Reciprocal Rank Fusion

最终结果排序(价格/销量/品牌等)

6.2 三路检索说明

6.3 RRF融合排序

RRF（Reciprocal Rank Fusion）算法：

• 公式：RRF(d) = Σ 1 / (k + rank(d))
• 参数 k 通常设为 60
• 将三路检索结果按排名倒数加权求和
• 最终按RRF得分降序排列

优势：

• 无需训练权重，简单有效
• 自动平衡不同检索路的结果
• 对排名靠前的结果给予更高权重

6.4 过滤与排序

支持的过滤条件：

• 平台过滤（京东/拼多多/1688/抖音）
• 价格区间过滤
• 品牌过滤
• 类目过滤
• 标签过滤
• 时间范围过滤

支持的排序方式：

• 相关度排序（默认，RRF得分）
• 价格升序/降序
• 销量排序
• 评分排序
• 上架时间排序

6.5 比价服务

比价逻辑：

比价结果包含：

• 各平台最低价
• 对应商品信息
• 平台商品数量
• 价格差异百分比
• 最优购买推荐

七、定时采集任务调度

7.1 任务调度架构

调度中心： XXL-JOB

#mermaid-svg-D46sPV5ltEBx4S1f{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-D46sPV5ltEBx4S1f .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-D46sPV5ltEBx4S1f .error-icon{fill:#552222;}#mermaid-svg-D46sPV5ltEBx4S1f .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-D46sPV5ltEBx4S1f .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-D46sPV5ltEBx4S1f .marker{fill:#333333;stroke:#333333;}#mermaid-svg-D46sPV5ltEBx4S1f .marker.cross{stroke:#333333;}#mermaid-svg-D46sPV5ltEBx4S1f svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-D46sPV5ltEBx4S1f p{margin:0;}#mermaid-svg-D46sPV5ltEBx4S1f .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-D46sPV5ltEBx4S1f .cluster-label text{fill:#333;}#mermaid-svg-D46sPV5ltEBx4S1f .cluster-label span{color:#333;}#mermaid-svg-D46sPV5ltEBx4S1f .cluster-label span p{background-color:transparent;}#mermaid-svg-D46sPV5ltEBx4S1f .label text,#mermaid-svg-D46sPV5ltEBx4S1f span{fill:#333;color:#333;}#mermaid-svg-D46sPV5ltEBx4S1f .node rect,#mermaid-svg-D46sPV5ltEBx4S1f .node circle,#mermaid-svg-D46sPV5ltEBx4S1f .node ellipse,#mermaid-svg-D46sPV5ltEBx4S1f .node polygon,#mermaid-svg-D46sPV5ltEBx4S1f .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-D46sPV5ltEBx4S1f .rough-node .label text,#mermaid-svg-D46sPV5ltEBx4S1f .node .label text,#mermaid-svg-D46sPV5ltEBx4S1f .image-shape .label,#mermaid-svg-D46sPV5ltEBx4S1f .icon-shape .label{text-anchor:middle;}#mermaid-svg-D46sPV5ltEBx4S1f .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-D46sPV5ltEBx4S1f .rough-node .label,#mermaid-svg-D46sPV5ltEBx4S1f .node .label,#mermaid-svg-D46sPV5ltEBx4S1f .image-shape .label,#mermaid-svg-D46sPV5ltEBx4S1f .icon-shape .label{text-align:center;}#mermaid-svg-D46sPV5ltEBx4S1f .node.clickable{cursor:pointer;}#mermaid-svg-D46sPV5ltEBx4S1f .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-D46sPV5ltEBx4S1f .arrowheadPath{fill:#333333;}#mermaid-svg-D46sPV5ltEBx4S1f .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-D46sPV5ltEBx4S1f .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-D46sPV5ltEBx4S1f .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-D46sPV5ltEBx4S1f .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-D46sPV5ltEBx4S1f .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-D46sPV5ltEBx4S1f .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-D46sPV5ltEBx4S1f .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-D46sPV5ltEBx4S1f .cluster text{fill:#333;}#mermaid-svg-D46sPV5ltEBx4S1f .cluster span{color:#333;}#mermaid-svg-D46sPV5ltEBx4S1f div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-D46sPV5ltEBx4S1f .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-D46sPV5ltEBx4S1f rect.text{fill:none;stroke-width:0;}#mermaid-svg-D46sPV5ltEBx4S1f .icon-shape,#mermaid-svg-D46sPV5ltEBx4S1f .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-D46sPV5ltEBx4S1f .icon-shape p,#mermaid-svg-D46sPV5ltEBx4S1f .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-D46sPV5ltEBx4S1f .icon-shape .label rect,#mermaid-svg-D46sPV5ltEBx4S1f .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-D46sPV5ltEBx4S1f .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-D46sPV5ltEBx4S1f .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-D46sPV5ltEBx4S1f :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

XXL-JOB 调度中心

任务管理界面- 任务配置/启停- 执行日志查看- 告警配置

执行器A(采集服务)

执行器B(清洗服务)

7.2 采集任务配置

7.3 任务执行流程

全量同步流程：

增量同步流程：

八、API接口设计

8.1 RESTful API列表

8.2 搜索接口说明

请求参数：

响应结构：

商品对象字段：

8.3 比价接口说明

请求参数：

响应结构：

九、部署架构

9.1 服务组件

9.3 资源配置建议

十、AI应用与核心算法

10.1 AI在系统中的角色

10.2 混合检索详细实现

10.2.1 检索流程总览

#mermaid-svg-vad6qgRX5MaZzPE2{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-vad6qgRX5MaZzPE2 .error-icon{fill:#552222;}#mermaid-svg-vad6qgRX5MaZzPE2 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-vad6qgRX5MaZzPE2 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-vad6qgRX5MaZzPE2 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-vad6qgRX5MaZzPE2 .marker.cross{stroke:#333333;}#mermaid-svg-vad6qgRX5MaZzPE2 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-vad6qgRX5MaZzPE2 p{margin:0;}#mermaid-svg-vad6qgRX5MaZzPE2 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 .cluster-label text{fill:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 .cluster-label span{color:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 .cluster-label span p{background-color:transparent;}#mermaid-svg-vad6qgRX5MaZzPE2 .label text,#mermaid-svg-vad6qgRX5MaZzPE2 span{fill:#333;color:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 .node rect,#mermaid-svg-vad6qgRX5MaZzPE2 .node circle,#mermaid-svg-vad6qgRX5MaZzPE2 .node ellipse,#mermaid-svg-vad6qgRX5MaZzPE2 .node polygon,#mermaid-svg-vad6qgRX5MaZzPE2 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-vad6qgRX5MaZzPE2 .rough-node .label text,#mermaid-svg-vad6qgRX5MaZzPE2 .node .label text,#mermaid-svg-vad6qgRX5MaZzPE2 .image-shape .label,#mermaid-svg-vad6qgRX5MaZzPE2 .icon-shape .label{text-anchor:middle;}#mermaid-svg-vad6qgRX5MaZzPE2 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-vad6qgRX5MaZzPE2 .rough-node .label,#mermaid-svg-vad6qgRX5MaZzPE2 .node .label,#mermaid-svg-vad6qgRX5MaZzPE2 .image-shape .label,#mermaid-svg-vad6qgRX5MaZzPE2 .icon-shape .label{text-align:center;}#mermaid-svg-vad6qgRX5MaZzPE2 .node.clickable{cursor:pointer;}#mermaid-svg-vad6qgRX5MaZzPE2 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-vad6qgRX5MaZzPE2 .arrowheadPath{fill:#333333;}#mermaid-svg-vad6qgRX5MaZzPE2 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-vad6qgRX5MaZzPE2 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-vad6qgRX5MaZzPE2 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-vad6qgRX5MaZzPE2 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-vad6qgRX5MaZzPE2 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-vad6qgRX5MaZzPE2 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-vad6qgRX5MaZzPE2 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-vad6qgRX5MaZzPE2 .cluster text{fill:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 .cluster span{color:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-vad6qgRX5MaZzPE2 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-vad6qgRX5MaZzPE2 rect.text{fill:none;stroke-width:0;}#mermaid-svg-vad6qgRX5MaZzPE2 .icon-shape,#mermaid-svg-vad6qgRX5MaZzPE2 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-vad6qgRX5MaZzPE2 .icon-shape p,#mermaid-svg-vad6qgRX5MaZzPE2 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-vad6qgRX5MaZzPE2 .icon-shape .label rect,#mermaid-svg-vad6qgRX5MaZzPE2 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-vad6qgRX5MaZzPE2 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-vad6qgRX5MaZzPE2 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-vad6qgRX5MaZzPE2 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

用户查询: iPhone 15 手机壳

Query处理(分词/纠错)

Query向量化(Qwen模型)

BM25检索(倒排索引)

稠密向量检索(HNSW索引)

稀疏向量检索(词元权重)

RRF 融合排序score = Σ 1/(k + rank_i(d))

业务规则过滤与重排(价格区间/品牌/平台/销量排序)

10.2.2 第一路：BM25关键字检索

实现原理：

• 基于ES倒排索引，使用ik分词器对中文分词
• BM25算法根据词频(TF)和逆文档频率(IDF)计算相关性

分词策略：

同义词扩展：

• 维护同义词词典：手机→移动电话、壳→保护套、iPhone→苹果手机
• 搜索时自动扩展同义词，提升召回

ES查询示例逻辑：

查询: "iPhone 15 手机壳"
分词: ["iPhone", "15", "手机壳", "手机", "壳"]
同义词扩展: ["iPhone", "苹果手机", "15", "手机壳", "保护套", "手机", "壳"]
BM25评分: 基于词频和逆文档频率计算相关性得分

优势与局限：

• 优势：精确匹配、可解释性强、速度快
• 局限：无法理解语义，“苹果手机壳"搜不到"iPhone保护套”

10.2.3 第二路：稠密向量语义检索

实现原理：

• 使用Qwen text-embedding-v3模型将查询和商品文本转为1024维向量
• 通过向量余弦相似度计算语义相关性
• ES使用HNSW近似最近邻索引加速检索

向量化流程：

#mermaid-svg-6nPyv03CdORlxZuz{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-6nPyv03CdORlxZuz .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-6nPyv03CdORlxZuz .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-6nPyv03CdORlxZuz .error-icon{fill:#552222;}#mermaid-svg-6nPyv03CdORlxZuz .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6nPyv03CdORlxZuz .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-6nPyv03CdORlxZuz .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6nPyv03CdORlxZuz .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6nPyv03CdORlxZuz .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-6nPyv03CdORlxZuz .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6nPyv03CdORlxZuz .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6nPyv03CdORlxZuz .marker{fill:#333333;stroke:#333333;}#mermaid-svg-6nPyv03CdORlxZuz .marker.cross{stroke:#333333;}#mermaid-svg-6nPyv03CdORlxZuz svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6nPyv03CdORlxZuz p{margin:0;}#mermaid-svg-6nPyv03CdORlxZuz .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-6nPyv03CdORlxZuz .cluster-label text{fill:#333;}#mermaid-svg-6nPyv03CdORlxZuz .cluster-label span{color:#333;}#mermaid-svg-6nPyv03CdORlxZuz .cluster-label span p{background-color:transparent;}#mermaid-svg-6nPyv03CdORlxZuz .label text,#mermaid-svg-6nPyv03CdORlxZuz span{fill:#333;color:#333;}#mermaid-svg-6nPyv03CdORlxZuz .node rect,#mermaid-svg-6nPyv03CdORlxZuz .node circle,#mermaid-svg-6nPyv03CdORlxZuz .node ellipse,#mermaid-svg-6nPyv03CdORlxZuz .node polygon,#mermaid-svg-6nPyv03CdORlxZuz .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-6nPyv03CdORlxZuz .rough-node .label text,#mermaid-svg-6nPyv03CdORlxZuz .node .label text,#mermaid-svg-6nPyv03CdORlxZuz .image-shape .label,#mermaid-svg-6nPyv03CdORlxZuz .icon-shape .label{text-anchor:middle;}#mermaid-svg-6nPyv03CdORlxZuz .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-6nPyv03CdORlxZuz .rough-node .label,#mermaid-svg-6nPyv03CdORlxZuz .node .label,#mermaid-svg-6nPyv03CdORlxZuz .image-shape .label,#mermaid-svg-6nPyv03CdORlxZuz .icon-shape .label{text-align:center;}#mermaid-svg-6nPyv03CdORlxZuz .node.clickable{cursor:pointer;}#mermaid-svg-6nPyv03CdORlxZuz .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-6nPyv03CdORlxZuz .arrowheadPath{fill:#333333;}#mermaid-svg-6nPyv03CdORlxZuz .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-6nPyv03CdORlxZuz .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-6nPyv03CdORlxZuz .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-6nPyv03CdORlxZuz .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-6nPyv03CdORlxZuz .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-6nPyv03CdORlxZuz .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-6nPyv03CdORlxZuz .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-6nPyv03CdORlxZuz .cluster text{fill:#333;}#mermaid-svg-6nPyv03CdORlxZuz .cluster span{color:#333;}#mermaid-svg-6nPyv03CdORlxZuz div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-6nPyv03CdORlxZuz .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-6nPyv03CdORlxZuz rect.text{fill:none;stroke-width:0;}#mermaid-svg-6nPyv03CdORlxZuz .icon-shape,#mermaid-svg-6nPyv03CdORlxZuz .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-6nPyv03CdORlxZuz .icon-shape p,#mermaid-svg-6nPyv03CdORlxZuz .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-6nPyv03CdORlxZuz .icon-shape .label rect,#mermaid-svg-6nPyv03CdORlxZuz .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-6nPyv03CdORlxZuz .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-6nPyv03CdORlxZuz .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-6nPyv03CdORlxZuz :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

用户搜索时

查询: 苹果手机保护套

Qwen Encoder

[0.11, -0.33, 0.55, …](查询向量)

ES HNSW向量检索返回Top-K相似商品

商品入库时

商品标题苹果iPhone 15 Pro Max手机壳

Qwen Encoder

[0.12, -0.34, 0.56, …](1024维稠密向量)

写入EStitle.dense_vector

相似度计算：

余弦相似度 = (A·B) / (||A|| × ||B||)

示例：
查询向量: "苹果手机保护套" → [0.11, -0.33, 0.55, …]
商品向量: "iPhone 15 Pro Max手机壳" → [0.12, -0.34, 0.56, …]

余弦相似度 = 0.92 (高度相似)
→ 即使字面不同，语义上匹配

HNSW索引参数：

优势与局限：

• 优势：语义理解强，能匹配字面不同但含义相同的商品
• 局限：计算开销大，对精确匹配不如BM25

10.2.4 第三路：稀疏向量词元检索（SPLADE）

实现原理：

• SPLADE模型将文本转为稀疏向量，每个维度对应词汇表中的一个词
• 向量值表示该词对文本的重要性权重
• 兼具BM25的精确性和向量的语义扩展能力

稀疏向量示例：

文本: "iPhone 15 手机壳"

SPLADE稀疏向量:
{
"iphone": 0.85,
"15": 0.72,
"手机": 0.68,
"手机壳": 0.91,
"保护套": 0.45, // 语义扩展
"苹果": 0.38, // 语义扩展
"配件": 0.25, // 语义扩展
…
}

与BM25的区别：

• BM25：基于统计，词权重由TF-IDF决定
• SPLADE：基于神经网络，自动学习词的重要性和扩展相关词

优势与局限：

• 优势：兼顾精确匹配和语义扩展，可解释性比稠密向量好
• 局限：依赖SPLADE模型质量，词汇表外词处理有限

10.2.5 RRF融合排序详解

RRF算法公式：

RRF_score(d) = Σ 1 / (k + rank_i(d))

其中：
– d: 文档（商品）
– k: 常数，通常设为60
– rank_i(d): 文档d在第i路检索中的排名

融合示例：

加权RRF（可选）：

RRF_score(d) = Σ weight_i / (k + rank_i(d))

默认权重：
– BM25: 0.3
– 稠密向量: 0.4
– 稀疏向量: 0.3

为什么k=60？

• k值影响排名靠前结果的权重差异
• k=60是信息检索领域的经验值
• k越小，排名靠前的结果权重差异越大
• k越大，各排名结果权重差异越平滑

10.2.6 检索性能优化

10.3 AI商品去重详细实现

10.3.1 去重场景

10.3.2 精确去重

规则：

唯一标识 = platform + product_id

去重逻辑：
1. 新数据到来时，检查 (platform, product_id) 是否已存在
2. 已存在 → 比较更新时间
– 新数据更新 → 覆盖旧数据，记录变更历史
– 旧数据更新 → 丢弃新数据
3. 不存在 → 直接入库

10.3.3 跨平台模糊去重（AI核心）

多维度相似度计算：

综合相似度 = w1 × 标题语义相似度
+ w2 × 品牌匹配度
+ w3 × 类目匹配度
+ w4 × 价格接近度

权重配置：
w1 = 0.40 (标题语义)
w2 = 0.25 (品牌匹配)
w3 = 0.20 (类目匹配)
w4 = 0.15 (价格接近)

（1）标题语义相似度

方法：向量余弦相似度

步骤：
1. 将两个商品标题分别转为1024维向量
2. 计算余弦相似度

示例：
标题A: "Apple iPhone 15 Pro Max 256GB 原色钛金属"
标题B: "苹果15ProMax 256G 原色钛金属手机"

向量A: [0.12, -0.34, 0.56, …]
向量B: [0.11, -0.33, 0.55, …]

余弦相似度 = 0.94 (高度相似)

（2）品牌匹配度

品牌别名映射表：

（3）类目匹配度

方法：类目树路径匹配

示例：
类目A: 手机通讯 > 手机 > 智能手机
类目B: 数码 > 手机 > 智能手机

匹配逻辑：
1. 计算类目路径的Jaccard相似度
2. 检查末级类目是否相同

Jaccard = |A∩B| / |A∪B| = 2/4 = 0.5
末级相同 → 额外加分

最终类目匹配度 = 0.7

（4）价格接近度

方法：基于价格比率计算

公式：
price_ratio = min(priceA, priceB) / max(priceA, priceB)
price_score = price_ratio (范围0-1)

示例：
价格A: 8999元
价格B: 8799元

price_ratio = 8799 / 8999 = 0.978
price_score = 0.978 (价格非常接近)

阈值：
– price_score >= 0.8: 价格接近
– 0.5 <= price_score < 0.8: 价格有一定差异
– price_score < 0.5: 价格差异大，可能不是同款

10.3.4 去重判定与处理

判定阈值：

同款商品合并策略：

10.3.5 去重性能优化

分层去重流程：

#mermaid-svg-npfqYAdhJUjxhfat{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-npfqYAdhJUjxhfat .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-npfqYAdhJUjxhfat .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-npfqYAdhJUjxhfat .error-icon{fill:#552222;}#mermaid-svg-npfqYAdhJUjxhfat .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-npfqYAdhJUjxhfat .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-npfqYAdhJUjxhfat .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-npfqYAdhJUjxhfat .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-npfqYAdhJUjxhfat .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-npfqYAdhJUjxhfat .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-npfqYAdhJUjxhfat .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-npfqYAdhJUjxhfat .marker{fill:#333333;stroke:#333333;}#mermaid-svg-npfqYAdhJUjxhfat .marker.cross{stroke:#333333;}#mermaid-svg-npfqYAdhJUjxhfat svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-npfqYAdhJUjxhfat p{margin:0;}#mermaid-svg-npfqYAdhJUjxhfat .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-npfqYAdhJUjxhfat .cluster-label text{fill:#333;}#mermaid-svg-npfqYAdhJUjxhfat .cluster-label span{color:#333;}#mermaid-svg-npfqYAdhJUjxhfat .cluster-label span p{background-color:transparent;}#mermaid-svg-npfqYAdhJUjxhfat .label text,#mermaid-svg-npfqYAdhJUjxhfat span{fill:#333;color:#333;}#mermaid-svg-npfqYAdhJUjxhfat .node rect,#mermaid-svg-npfqYAdhJUjxhfat .node circle,#mermaid-svg-npfqYAdhJUjxhfat .node ellipse,#mermaid-svg-npfqYAdhJUjxhfat .node polygon,#mermaid-svg-npfqYAdhJUjxhfat .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-npfqYAdhJUjxhfat .rough-node .label text,#mermaid-svg-npfqYAdhJUjxhfat .node .label text,#mermaid-svg-npfqYAdhJUjxhfat .image-shape .label,#mermaid-svg-npfqYAdhJUjxhfat .icon-shape .label{text-anchor:middle;}#mermaid-svg-npfqYAdhJUjxhfat .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-npfqYAdhJUjxhfat .rough-node .label,#mermaid-svg-npfqYAdhJUjxhfat .node .label,#mermaid-svg-npfqYAdhJUjxhfat .image-shape .label,#mermaid-svg-npfqYAdhJUjxhfat .icon-shape .label{text-align:center;}#mermaid-svg-npfqYAdhJUjxhfat .node.clickable{cursor:pointer;}#mermaid-svg-npfqYAdhJUjxhfat .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-npfqYAdhJUjxhfat .arrowheadPath{fill:#333333;}#mermaid-svg-npfqYAdhJUjxhfat .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-npfqYAdhJUjxhfat .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-npfqYAdhJUjxhfat .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-npfqYAdhJUjxhfat .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-npfqYAdhJUjxhfat .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-npfqYAdhJUjxhfat .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-npfqYAdhJUjxhfat .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-npfqYAdhJUjxhfat .cluster text{fill:#333;}#mermaid-svg-npfqYAdhJUjxhfat .cluster span{color:#333;}#mermaid-svg-npfqYAdhJUjxhfat div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-npfqYAdhJUjxhfat .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-npfqYAdhJUjxhfat rect.text{fill:none;stroke-width:0;}#mermaid-svg-npfqYAdhJUjxhfat .icon-shape,#mermaid-svg-npfqYAdhJUjxhfat .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-npfqYAdhJUjxhfat .icon-shape p,#mermaid-svg-npfqYAdhJUjxhfat .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-npfqYAdhJUjxhfat .icon-shape .label rect,#mermaid-svg-npfqYAdhJUjxhfat .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-npfqYAdhJUjxhfat .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-npfqYAdhJUjxhfat .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-npfqYAdhJUjxhfat :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

否

是

相似度>=0.85

0.70<=相似度<0.85

相似度<0.70

新商品到来

第一层：粗筛品牌相同? 类目相同? 价格接近?

通过?

独立入库

第二层：向量召回标题向量检索Top-10候选

第三层：精判多维度相似度计算综合得分判定

判定结果

合并数据

标记待审核

独立入库

10.4 混合检索调优

10.5 数据更新策略

10.6 性能优化

十一、风险与应对

文档版本: v1.0
创建日期: 2026-06-11
最后更新: 2026-06-11