C++实现Boost搜索引擎数据清洗全解析

好的，我们来详细讲解C++实现的基于正倒排索引的Boost搜索引擎项目中的数据清洗模块（上篇）。数据清洗是搜索引擎预处理的关键步骤，直接影响后续索引构建的质量。

一、数据清洗的核心目标

二、核心代码实现

1. HTML标签剥离

#include <regex>
#include <string>

// 使用正则表达式移除HTML标签
std::string remove_html_tags(const std::string& raw_content) {
// 匹配尖括号内的任意内容（包括注释）
std::regex html_pattern(R"(<[^>]*>)");
return std::regex_replace(raw_content, html_pattern, "");
}

https://weibo.com/tv/show/1034:5275492375527436
https://weibo.com/tv/show/1034:5275492341973002
https://weibo.com/tv/show/1034:5275492300029959
https://weibo.com/tv/show/1034:5275492257824786
https://weibo.com/tv/show/1034:5275492207755286
https://weibo.com/tv/show/1034:5275492170006540
https://weibo.com/tv/show/1034:5275492128063495
https://weibo.com/tv/show/1034:5275492090052611
https://weibo.com/tv/show/1034:5275492056760333
https://weibo.com/tv/show/1034:5275492018749442
https://weibo.com/tv/show/1034:5275491968679948
https://weibo.com/tv/show/1034:5275491922280454
https://weibo.com/tv/show/1034:5275491880337439
https://weibo.com/tv/show/1034:5275491834462235
https://weibo.com/tv/show/1034:5275491796713479

2. 字符编码归一化

#include <iconv.h>
#include <stdexcept>

std::string convert_to_utf8(const std::string& input, const char* from_encoding) {
iconv_t cd = iconv_open("UTF-8", from_encoding);
if (cd == (iconv_t)-1) {
throw std::runtime_error("Encoding conversion failed");
}

size_t in_bytes = input.size();
size_t out_bytes = in_bytes * 4; // UTF-8最大4字节/字符
std::string output(out_bytes, '\\0');

char* in_ptr = const_cast<char*>(input.data());
char* out_ptr = output.data();

if (iconv(cd, &in_ptr, &in_bytes, &out_ptr, &out_bytes) == (size_t)-1) {
iconv_close(cd);
throw std::runtime_error("Conversion error");
}

iconv_close(cd);
output.resize(output.size() – out_bytes); // 调整实际大小
return output;
}

3. 正文提取与噪声过滤

// 示例：提取<title>标签内容
std::string extract_title(const std::string& html_content) {
std::regex title_regex(R"(<title>(.*?)</title>)", std::regex::icase);
std::smatch match;
if (std::regex_search(html_content, match, title_regex) && match.size() > 1) {
return remove_html_tags(match[1].str());
}
return "Untitled";
}

// 基于统计的正文定位（简化版）
std::string extract_main_content(const std::string& cleaned_content) {
// 实际项目需使用基于标签密度/文本密度的算法
// 此处仅展示跳过头部/尾部噪声的思路
const size_t header_threshold = 200;
const size_t footer_threshold = 100;
size_t len = cleaned_content.length();

if (len <= header_threshold + footer_threshold)
return cleaned_content;

return cleaned_content.substr(
header_threshold,
len – header_threshold – footer_threshold
);
}

https://weibo.com/tv/show/1034:5275492375527436
https://weibo.com/tv/show/1034:5275492341973002
https://weibo.com/tv/show/1034:5275492300029959
https://weibo.com/tv/show/1034:5275492257824786
https://weibo.com/tv/show/1034:5275492207755286
https://weibo.com/tv/show/1034:5275492170006540
https://weibo.com/tv/show/1034:5275492128063495
https://weibo.com/tv/show/1034:5275492090052611
https://weibo.com/tv/show/1034:5275492056760333
https://weibo.com/tv/show/1034:5275492018749442
https://weibo.com/tv/show/1034:5275491968679948
https://weibo.com/tv/show/1034:5275491922280454
https://weibo.com/tv/show/1034:5275491880337439
https://weibo.com/tv/show/1034:5275491834462235
https://weibo.com/tv/show/1034:5275491796713479

三、关键问题详解

1. 正则表达式效率优化

• 预编译正则：将常用正则表达式声明为static const避免重复编译

  static const std::regex g_html_tag_re(R"(<[^>]*>)");

• 避免贪婪匹配：使用.*?非贪婪匹配防止性能退化

2. 编码转换注意事项

• BOM头处理：UTF-8文档可能包含EF BB BF头，需手动移除

  if (output.size() >= 3 &&
  static_cast<unsigned char>(output[0]) == 0xEF &&
  static_cast<unsigned char>(output[1]) == 0xBB &&
  static_cast<unsigned char>(output[2]) == 0xBF) {
  output.erase(0, 3);
  }

• 错误回退机制：对无法转换的字符使用//TRANSLIT模式（需iconv支持）

3. 内存安全实践

• 缓冲区预留：std::string转换时预留4倍空间避免溢出
• 异常安全：使用RAII管理iconv_t描述符

  class IConvWrapper {
  public:
  IConvWrapper(const char* to, const char* from)
  : cd_(iconv_open(to, from)) {
  if (cd_ == (iconv_t)-1) throw…;
  }
  ~IConvWrapper() { if (cd_ != (iconv_t)-1) iconv_close(cd_); }
  // 转换操作封装…
  private:
  iconv_t cd_;
  };

四、测试用例设计

void test_html_removal() {
std::string html = "<div>Hello<br/><script>alert();</script></div>";
std::string cleaned = remove_html_tags(html);
assert(cleaned == "Hello");
}

void test_encoding_conversion() {
std::string gbk_str = "\\xB0\\xD9\\xB6\\xC8"; // "百度"的GBK编码
std::string utf8_str = convert_to_utf8(gbk_str, "GBK");
assert(utf8_str == u8"百度"); // C++11 u8前缀
}

五、下篇预告

下一篇我们将深入讲解：

注：实际项目中需根据网页特征调整参数，建议使用成熟的库如Gumbo解析HTML。本文展示的是核心逻辑的简化实现。