filebeat to kafka client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout #filebeat #42346

871690991 · 2025-01-20T03:21:02Z

my filebeat version 7.6.0 ,kafka version3.2.0

I found that Filebeat exceeded the time limit when going to Kafka. Every time I checked Kafka, it was normal. I used a script to write Kafka data, which was normal. However, every time this problem occurred, I had to restart it. After a few days, it reappeared

`2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-02-prd.bjm5.topsports.lan:9092
2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [ops-systemservice-logs] from broker ops-elk-kafka-01-prd.prd.lan:9092

2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-02-prd.bjm5.topsports.lan:9092
2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [member-appbiz-logs] from broker ops-elk-kafka-01-prd.prd.lan:9092

2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-02-prd.bjm5.topsports.lan:9092
2025-01-20T09:39:55.270+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [ops-systemservice-logs] from broker ops-elk-kafka-01-prd.prd.lan:9092

2025-01-20T09:39:55.519+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [member-appsensors-logs] from broker ops-elk-kafka-01-prd.prd.lan:9092

2025-01-20T09:39:57.954+0800 INFO log/input.go:576 File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /data/app/logs/fe-wxwork-center/sensors/msd_backend/service_log_prd_lan_10.123.87.137.2025-01-20
2025-01-20T09:40:07.955+0800 INFO log/input.go:576 File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /data/app/logs/fe-wxwork-center/sensors/msd_backend/service_log_prd_lan_10.123.87.137.2025-01-20
2025-01-20T09:40:14.957+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1070070,"time":{"ms":15}},"total":{"ticks":8385300,"time":{"ms":43},"value":8385300},"user":{"ticks":7315230,"time":{"ms":28}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":576},"info":{"ephemeral_id":"3124b90c-6781-444b-a54c-4ff9ac26de2a","uptime":{"ms":258660027}},"memstats":{"gc_next":86119040,"memory_alloc":53009088,"memory_total":1053865185992},"runtime":{"goroutines":3524}},"filebeat":{"harvester":{"files":{"54465d74-1a31-48d9-a6c6-faf73b991976":{"size":214},"89193983-7963-4caa-8ed0-f878bbd87cd4":{"size":54224},"a4614eaa-825a-4595-b75a-99e82f165800":{"size":7690}},"open_files":568,"running":565}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":7,"events":{"active":4122}}},"registrar":{"states":{"current":973}},"system":{"load":{"1":1.75,"15":2.59,"5":2.32,"norm":{"1":0.0273,"15":0.0405,"5":0.0363}}}}}}
2025-01-20T09:40:17.956+0800 INFO log/input.go:576 File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /data/app/logs/fe-wxwork-center/sensors/msd_backend/service_log_prd_lan_10.123.87.137.2025-01-20
2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 Failed to connect to broker ops-elk-kafka-01-prd.prd.lan:9092: dial tcp: i/o timeout

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [member-gatewayaccess-logs] from broker ops-elk-kafka-03-prd.bjm5.topsports.lan:9092

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-01-prd.prd.lan:9092
2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [member-appbiz-logs] from broker ops-elk-kafka-03-prd.bjm5.topsports.lan:9092

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-01-prd.prd.lan:9092
2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [member-appsensors-logs] from broker ops-elk-kafka-03-prd.bjm5.topsports.lan:9092

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-01-prd.prd.lan:9092
2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [ops-systemservice-logs] from broker ops-elk-kafka-03-prd.bjm5.topsports.lan:9092

2025-01-20T09:40:25.269+0800 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp: i/o timeout

2025-01-20T09:40:25.270+0800 INFO kafka/log.go:53 client/brokers deregistered broker #-1 at ops-elk-kafka-01-prd.prd.lan:9092
2025-01-20T09:40:25.270+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [member-gatewayaccess-logs] from broker ops-elk-kafka-03-prd.bjm5.topsports.lan:9092