Zabbix对Kafka topic积压数据监控的问题(bug优化)

一   自动分区

1.1  优化前计算方式

寻找配置文件

vim consumer-groups.conf

写入配置文件

test-group|test

执行脚本

bash consumer-groups.sh discovery
{
"data": [
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"0" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"1" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"3" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"2" }
]
}

由上可知，我们只有test-group|test这一个自动发现配置文件是没有问题的。然后接入test-group|test1

1.2  未优化前计算方式

寻找配置文件

vim consumer-groups.conf

写入配置文件

test-group|test
test-group|test1

执行脚本

bash consumer-groups.sh discovery
{
"data": [
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"0" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"1" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"3" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"2" }
{ "{#GROUP}":"test-group", "{#TOPICP}":"test1", "{#PARTITION}":"0" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test2", "{#PARTITION}":"1" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test3", "{#PARTITION}":"2" }
]
}

执行完我们发现，上面这种种格式是不对的，会导致我们的监控项会出现问题

1.3  优化计算方式

寻找配置文件

vim consumer-groups.conf

写入配置文件

test-group|test
test-group|test1

执行脚本

bash consumer-groups.sh discovery
{
"data": [
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"0" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"1" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"3" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test", "{#PARTITION}":"2" }，
{ "{#GROUP}":"test-group", "{#TOPICP}":"test1", "{#PARTITION}":"0" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test1", "{#PARTITION}":"1" },
{ "{#GROUP}":"test-group", "{#TOPICP}":"test1", "{#PARTITION}":"2" }
]
}

1.4  lag分区

优化后计算方式

# test-group test分区0 lag
bash consumer-groups.sh lag test-group test 0

# test-group test分区1 lag
bash consumer-groups.sh lag test-group test 1

# test-group test1分区0 lag
bash consumer-groups.sh lag test-group test1 0

我们与未优化的计算方式对比下

优化前计算方式

# 获取分区0 lag
bash consumer-groups.sh lag 0

# 获取分区1 lag
bash consumer-groups.sh lag 1

# 获取分区2 lag
bash consumer-groups.sh lag 2

# 获取分区3 lag
bash consumer-groups.sh lag 3

最终优化后脚本

vim consumer-groups.conf
test-group|test
test-group|test1
vim consumer-groups.sh
cal_topic() {
if [ $# -ne 2 ]; then
echo "parameter num error, 读取topic信息失败"
exit 1
else
/usr/local/kafka/bin/./kafka-consumer-groups.sh --bootstrap-server 192.168.3.55:9092 --describe --group $1 |grep -w $2|grep -v none
fi
}
topic_discovery() {
printf "{\n"
printf "\t\"data\": [\n"
m=0
num=`cat /etc/zabbix/monitor_scripts/consumer-groups.conf|wc -l`
for line in `cat /etc/zabbix/monitor_scripts/consumer-groups.conf`
do
m=`expr $m + 1`
group=`echo ${line} | awk -F'|' '{print $1}'`
topic=`echo ${line} | awk -F'|' '{print $2}'`
cal_topic $group $topic > /tmp/consumer-group-tmp
count=`cat /tmp/consumer-group-tmp|wc -l`
n=0
while read line
do
n=`expr $n + 1`
if [ $n -eq $count ] && [ $m -eq $num ]; then
topicp=`echo $line | awk '{print $1}'`
partition=`echo $line | awk '{print $2}'`
printf "\t\t{ \"{#GROUP}\":\"${group}\", \"{#TOPICP}\":\"${topicp}\", \"{#PARTITION}\":\"${partition}\" }\n"
else
topicp=`echo $line | awk '{print $1}'`
partition=`echo $line | awk '{print $2}'`
printf "\t\t{ \"{#GROUP}\":\"${group}\", \"{#TOPICP}\":\"${topicp}\", \"{#PARTITION}\":\"${partition}\" },\n"
fi
done < /tmp/consumer-group-tmp
done
printf "\t]\n"
printf "}\n"
}
if [ $1 == "discovery" ]; then
topic_discovery
elif [ $1 == "lag" ];then
cal_topic $2 $3 > /tmp/consumer-group
cat /tmp/consumer-group |awk -v t=$3 -v p=$4 '{if($1==t && $2==p ){print $5}}'
else
echo "Usage: /data/scripts/consumer-group.sh discovery | lag"
fi
bash consumer-groups.sh discovery
## test-group test分区0 lag
bash consumer-groups.sh lag test-group test 0

二  Zabbix接入

2.1  Zabbix配置文件

vim userparameter_kafka.conf
UserParameter=topic_discovery,bash /data/scripts/consumer-groups.sh discovery
UserParameter=topic_log[*],bash /data/scripts/consumer-groups.sh lag "$1" "$2" "$3"

2.2  Zabbix

2.3  配置监控项

2.4  告警信息

告警主机：Kafka_192.168.3.55
主机IP：192.168.3.55
主机组：Kafka
告警时间：2022.03.21 00:23:10
告警等级：Average
告警信息：test-group/test/分区1:数据积压100
告警项目：topic_lag[test-group,test,1]