【异常解决】The coordinator is not available

问题

最近上线跑了一个flink任务，运行不久，就会挂掉，初步查看日志报错如下

WARN  org.apache.flink.connector.kafka.source.reader.KafkaSourceReader [] - Failed to commit consumer offsets for checkpoint 1
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.
Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: The coordinator is not available.

报错原因为The coordinator is not available.

报错在网上搜了一下，根据网友的经验，是消费组协调leader不存在导致

查看kafka __consumer_offsets topic

kafka-topics.sh -bootstrap-server node1:9092,node2:9092,node3:9092,node4:9092,node5:9092 --topic __consumer_offsets --describe

发现确实有Leader:none的

可能原因是 __consumer_offset topic的默认分区是50，但是备份只有1份; kafka集群部署了5台，也就是5个brokers

消费者组连接kafka，并会请求某一台来查找Coordinator(协调者)，如果连接的机器上没有备份就不会有Leader,就会出现

找不到Coordinator(协调者)

查看offsets.topic.replication.factor默认配置

cat kafak/config/server.properties | grep offsets.topic.replication.factor

默认值果然只是1；

解决方式

• 1、停止kafka, 修改kafka配置 config/server.properties添加

  修改成broker的数量

  offsets.topic.replication.factor=3
  

  • 2、删除zookeeper配置信息

    ./bin/zkCli.sh -server 127.0.0.1:2181
    

    • 删除/config/topics/__consumer_offsets

      delete /config/topics/__consumer_offsets
      

      • 删除/brokers/topics/__consumer_offsets

        deleteall /brokers/topics/__consumer_offsets
        

        • 3、重启kafka