欢迎光临
我们一直在努力

Kafka 与 Schema Registry:实现高效的序列化治理与 Schema 演进

  • Kafka 序列化挑战与 Schema Registry 概述
  • 在分布式消息系统中,Kafka 作为高性能的消息中间件被广泛应用。然而,随着系统规模扩大和数据类型复杂化,消息序列化逐渐成为挑战。传统的序列化方法如 JSON 虽然灵活但缺乏类型安全与版本控制,而二进制格式如 Avro 和 Protobuf 虽然高效但面临 Schema 版本管理问题。

    Schema Registry 作为 Confluent 平台的核心组件,提供了集中化的 Schema 管理解决方案。它充当 Schema 的中央存储库,并与 Kafka 集成,实现了生产者和消费者之间的 Schema 共享与验证。通过 Schema Registry,我们可以确保消息格式的一致性,并在 Schema 演进时提供兼容性检查。

  • Avro 与 Protobuf 序列化机制对比
  • Avro 和 Protobuf 是两种广泛应用于 Kafka 的序列化格式,它们各自具有特点和优势。

    | 特性 | Avro | Protobuf |

    |——|——|———|

    | 数据格式 | 基于文本的 Schema 定义 | 基于文本的 Schema 定义 |

    | 序列化结果 | 二进制格式,紧凑高效 | 二进制格式,非常紧凑 |

    | Schema 演进 | 支持向后兼容,字段可添加/删除 | 支持向前和向后兼容 |

    | 代码生成 | 运行时或编译时生成 | 编译时生成 |

    | 性能 | 序列化速度快,解析速度一般 | 序列化和解析速度都非常快 |

    | 兼容性检查 | Schema Registry 自动验证 | 需要手动或通过工具验证 |

    Avro 的 Schema 定义使用 JSON 格式,而 Protobuf 使用 .proto 文件定义。Avro 在 Schema 演进方面提供了更为灵活的机制,特别适合动态变化的数据模型。Protobuf 则以其高效的二进制编码和强大的编译时类型检查著称,适合性能要求高的场景。

  • Schema 演进机制与最佳实践
  • Schema 演进是 Schema Registry 的核心功能之一,它允许在不破坏现有消费者的情况下更新数据格式。以下是 Schema 演进的基本规则和最佳实践:

    Avro Schema 演进规则:

    • 可以添加新字段,必须设置默认值
    • 可以删除字段,但字段必须已被标记为 deprecated
    • 修改字段的类型必须保持兼容(如 int 可转为 long,但不能转为 string)
    • 不能修改字名的顺序(除非使用 "field_name" 显式指定)

    Protobuf Schema 演进规则:

    • 可以添加新字段,新字段编号不能重复使用已删除字段的编号
    • 可以删除字段,但不能重用其编号
    • 可以修改字段类型,但要保持兼容(如 int32 可转为 int64,但不能转为 string)
    • 可以将 optional 字段改为 required,但不能反向操作

    Schema 演进最佳实践:

  • 为每个主题使用独立的 Schema Registry
  • 始终向后兼容设计新字段
  • 保留字段编号范围,为未来扩展预留空间
  • 使用语义版本控制 Schema 变更
  • 在生产环境变更前进行充分测试
  • 实战案例:最小化配置与运行示例
  • 下面是一个使用 Avro 和 Schema Registry 的最小配置示例:

    // 生产者配置
    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("key.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
    props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
    props.put("schema.registry.url", "http://localhost:8081");
    // 创建 Schema Registry 客户端
    SchemaRegistryClient schemaRegistry = new CachedSchemaRegistryClient(
    "http://localhost:8081", 100);
    // 定义 Avro Schema
    String schemaString = "{\\"type\\":\\"record\\",\\"name\\":\\"User\\",\\"fields\\":[{\\"name\\":\\"name\\",\\"type\\":\\"string\\"},{\\"name\\":\\"age\\",\\"type\\":[\\"null\\",\\"int\\"],\\"default\\":null}]}";
    Schema schema = new Schema.Parser().parse(schemaString);
    // 注册 Schema
    int id = schemaRegistry.register("user-topic-value", schema);
    // 创建生产者
    KafkaProducer<String, GenericRecord> producer = new KafkaProducer<>(props);
    // 创建记录并发送
    GenericRecord record = new GenericRecord(schema);
    record.put("name", "Alice");
    record.put("age", 30);
    ProducerRecord<String, GenericRecord> producerRecord =
    new ProducerRecord<>("user-topic", null, record);
    producer.send(producerRecord);
    producer.close();
    // 消费者配置
    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("key.deserializer", "io.confluent.kafka.serializers.KafkaAvroDeserializer");
    props.put("value.deserializer", "io.confluent.kafka.serializers.KafkaAvroDeserializer");
    props.put("schema.registry.url", "http://localhost:8081");
    props.put("specific.avro.reader", "true");
    // 创建消费者
    KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<>(props);
    consumer.subscribe(Collections.singletonList("user-topic"));
    // 消费消息
    ConsumerRecords<String, GenericRecord> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, GenericRecord> record : records) {
    System.out.println("Received message: " + record.value());
    }
    consumer.close();

    注意事项:

  • 确保 Schema Registry 和 Kafka 集群正常运行
  • 添加 Confluent 客户端依赖到项目中
  • 在生产环境使用时,考虑添加重试机制和错误处理
  • 监控 Schema Registry 的健康状态和性能指标
  • 定期备份 Schema 数据以防止丢失
  • #publish-mermaid-1788402409874-0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#publish-mermaid-1788402409874-0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#publish-mermaid-1788402409874-0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#publish-mermaid-1788402409874-0 .error-icon{fill:#552222;}#publish-mermaid-1788402409874-0 .error-text{fill:#552222;stroke:#552222;}#publish-mermaid-1788402409874-0 .edge-thickness-normal{stroke-width:1px;}#publish-mermaid-1788402409874-0 .edge-thickness-thick{stroke-width:3.5px;}#publish-mermaid-1788402409874-0 .edge-pattern-solid{stroke-dasharray:0;}#publish-mermaid-1788402409874-0 .edge-thickness-invisible{stroke-width:0;fill:none;}#publish-mermaid-1788402409874-0 .edge-pattern-dashed{stroke-dasharray:3;}#publish-mermaid-1788402409874-0 .edge-pattern-dotted{stroke-dasharray:2;}#publish-mermaid-1788402409874-0 .marker{fill:#333333;stroke:#333333;}#publish-mermaid-1788402409874-0 .marker.cross{stroke:#333333;}#publish-mermaid-1788402409874-0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#publish-mermaid-1788402409874-0 p{margin:0;}#publish-mermaid-1788402409874-0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#publish-mermaid-1788402409874-0 .cluster-label text{fill:#333;}#publish-mermaid-1788402409874-0 .cluster-label span{color:#333;}#publish-mermaid-1788402409874-0 .cluster-label span p{background-color:transparent;}#publish-mermaid-1788402409874-0 .label text,#publish-mermaid-1788402409874-0 span{fill:#333;color:#333;}#publish-mermaid-1788402409874-0 .node rect,#publish-mermaid-1788402409874-0 .node circle,#publish-mermaid-1788402409874-0 .node ellipse,#publish-mermaid-1788402409874-0 .node polygon,#publish-mermaid-1788402409874-0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788402409874-0 .rough-node .label text,#publish-mermaid-1788402409874-0 .node .label text,#publish-mermaid-1788402409874-0 .image-shape .label,#publish-mermaid-1788402409874-0 .icon-shape .label{text-anchor:middle;}#publish-mermaid-1788402409874-0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#publish-mermaid-1788402409874-0 .rough-node .label,#publish-mermaid-1788402409874-0 .node .label,#publish-mermaid-1788402409874-0 .image-shape .label,#publish-mermaid-1788402409874-0 .icon-shape .label{text-align:center;}#publish-mermaid-1788402409874-0 .node.clickable{cursor:pointer;}#publish-mermaid-1788402409874-0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#publish-mermaid-1788402409874-0 .arrowheadPath{fill:#333333;}#publish-mermaid-1788402409874-0 .edgePath .path{stroke:#333333;stroke-width:1px;}#publish-mermaid-1788402409874-0 .flowchart-link{stroke:#333333;fill:none;}#publish-mermaid-1788402409874-0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788402409874-0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#publish-mermaid-1788402409874-0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788402409874-0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#publish-mermaid-1788402409874-0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#publish-mermaid-1788402409874-0 .cluster text{fill:#333;}#publish-mermaid-1788402409874-0 .cluster span{color:#333;}#publish-mermaid-1788402409874-0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#publish-mermaid-1788402409874-0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#publish-mermaid-1788402409874-0 rect.text{fill:none;stroke-width:0;}#publish-mermaid-1788402409874-0 .icon-shape,#publish-mermaid-1788402409874-0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788402409874-0 .icon-shape p,#publish-mermaid-1788402409874-0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#publish-mermaid-1788402409874-0 .icon-shape .label rect,#publish-mermaid-1788402409874-0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788402409874-0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#publish-mermaid-1788402409874-0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#publish-mermaid-1788402409874-0 .node .neo-node{stroke:#9370DB;}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node rect,#publish-mermaid-1788402409874-0 [data-look=\”neo\”].cluster rect,#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node polygon{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].swimlane.cluster rect{filter:none;}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node path{stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node .outer-path{filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node .neo-line path{stroke:#9370DB;filter:none;}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node circle{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].node circle .state-start{fill:#000000;}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].icon-shape .icon{fill:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788402409874-0 [data-look=\”neo\”].icon-shape .icon-neo path{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788402409874-0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

    生产者创建消息

    序列化器引用 Schema

    检查 Schema Registry 中是否存在

    不存在:注册新 Schema

    获取 Schema ID

    存在:获取已有 Schema ID

    序列化消息并带上 Schema ID

    发送消息至 Kafka

    消费者接收消息

    反序列化器使用 Schema ID

    从 Registry 获取 Schema

    验证消息 Schema

    成功:反序列化消息

    失败:抛出异常

    赞(0)
    未经允许不得转载:171主机测评 » Kafka 与 Schema Registry:实现高效的序列化治理与 Schema 演进
    分享到: 更多 (0)

    评论 抢沙发

    • 昵称 (必填)
    • 邮箱 (必填)
    • 网址