共计 10280 个字符,预计需要花费 26 分钟才能阅读完成。
最近在搞 Kafka 集群监控,之前也是看了网上的很多资料。之所以使用 jmxtrans+influxdb+grafana 是因为界面酷炫,可以定制化,缺点是不能操作 Kafka 集群,可能需要配合 Kafka Manager 一起使用。
环境信息
CentOS Linux release 7.6.1810 (Core)
jdk1.8.0_201
zookeeper-3.4.14
kafka_2.11-2.2.0
开启 Kafka JMX 端口
JMX(Java Management Extensions,即 Java 管理扩展)是一个为应用程序、设备、系统等植入管理功能的框架。JMX 可以跨越一系列异构操作系统平台、系统体系结构和网络传输协议,灵活的开发无缝集成的系统、网络和服务管理应用。Kafka 做为一款 Java 应用,已经定义了丰富的性能指标,(可以参考 Kafka 监控指标),通过 JMX 可以轻松对其进行监控。
在 ${KAFKA_HOME}/bin/ 路径下修改 kafka-run-class.sh 脚本,第一行增加 JMX_PORT=9999 即可。
JMX_PORT=9999
重启 Kafka
./bin/kafka-server-stop.sh
./bin/kafka-server-start.sh -daemon ./config/server.properties
重启后查看 Kafka 以及 JMX 端口状态
ps -ef | grep kafka
root 8273 1 99 02:32 pts/0 00:00:09 /opt/jdk1.8.0_201/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 …… kafka.Kafka ./config/server.properties
netstat -anop | grep 9999
tcp6 0 0 :::9999 :::* LISTEN 8273/java off (0.00/0/0)
安装 InfluxDB
InfluxDB 是一个时间序列数据库,用于处理海量写入与负载查询。InfluxDB 旨在用作涉及大量时间戳数据的任何用例(包括 DevOps 监控,应用程序指标,物联网传感器数据和实时分析)的后端存储。
下载 InfluxDB rpm 安装包
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.7.5.x86_64.rpm
–2019-04-10 02:52:30– https://dl.influxdata.com/influxdb/releases/influxdb-1.7.5.x86_64.rpm
Resolving dl.influxdata.com (dl.influxdata.com)… 54.192.151.21, 54.192.151.81, 54.192.151.87, …
Connecting to dl.influxdata.com (dl.influxdata.com)|54.192.151.21|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 46536692 (44M) [application/octet-stream]
Saving to:‘influxdb-1.7.5.x86_64.rpm’
100%[================================================================================================================================================================================>] 46,536,692 440KB/s in 60s
2019-04-10 02:53:37 (756 KB/s) –‘influxdb-1.7.5.x86_64.rpm’saved [46536692/46536692]
安装 rpm 包
rpm -ivh influxdb-1.7.5.x86_64.rpm
Preparing… ################################# [100%]
Updating / installing…
1:influxdb-1.7.5-1 ################################# [100%]
Created symlink from /etc/systemd/system/influxd.service to /usr/lib/systemd/system/influxdb.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/influxdb.service to /usr/lib/systemd/system/influxdb.service.
启动 InfluxDB
service influxdb start
Redirecting to /bin/systemctl start influxdb.service
查看 InfluxDB 状态
ps -ef | grep influxdb
influxdb 8475 1 2 03:01 ? 00:00:00 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
root 8486 7007 0 03:02 pts/0 00:00:00 grep –color=auto influxdb
service influxdb status
Redirecting to /bin/systemctl status influxdb.service
● influxdb.service – InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/usr/lib/systemd/system/influxdb.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-04-10 03:01:48 EDT; 22s ago
Docs: https://docs.influxdata.com/influxdb/
Main PID: 8475 (influxd)
CGroup: /system.slice/influxdb.service
└─8475 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375804Z lvl=info msg=”Starting precreation service” log_id=0EiWgWRl000 service=shard-precreation check_interval=10m advance_period=30m
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375810Z lvl=info msg=”Starting snapshot service” log_id=0EiWgWRl000 service=snapshot
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375816Z lvl=info msg=”Starting continuous query service” log_id=0EiWgWRl000 service=continuous_querier
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375826Z lvl=info msg=”Starting HTTP service” log_id=0EiWgWRl000 service=httpd authentication=false
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375830Z lvl=info msg=”opened HTTP access log” log_id=0EiWgWRl000 service=httpd path=stderr
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375936Z lvl=info msg=”Listening on HTTP” log_id=0EiWgWRl000 service=httpd addr=[::]:8086 https=false
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375949Z lvl=info msg=”Starting retention policy enforcement service” log_id=0EiWgWRl000 service=retention check_interval=30m
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.376138Z lvl=info msg=”Listening for signals” log_id=0EiWgWRl000
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.376389Z lvl=info msg=”Storing statistics” log_id=0EiWgWRl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.376534Z lvl=info msg=”Sending usage statistics to usage.influxdata.com” log_id=0EiWgWRl000
使用 InfluxDB 客户端
influx
Connected to http://localhost:8086 version 1.7.5
InfluxDB shell version: 1.7.5
Enter an InfluxQL query
>
创建用户和数据库
> CREATE USER “admin” WITH PASSWORD ‘admin’ WITH ALL PRIVILEGES
> create database “jmxDB”
创建完成 InfluxDB 的用户和数据库暂时就够用了,其它简单操作如下,后面会用到
# 创建数据库
create database “db_name”
# 显示所有的数据库
show databases
# 删除数据库
drop database “db_name”
# 使用数据库
use db_name
# 显示该数据库中所有的表
show measurements
# 创建表,直接在插入数据的时候指定表名
insert test,host=127.0.0.1,monitor_name=test count=1
# 删除表
drop measurement “measurement_name”
# 退出
quit
安装 jmxtrans
jmxtrans 的作用是自动去 jvm 中获取所有 jmx 格式数据,并按照某种格式(json 文件配置格式)输出到其他应用程序(本例中的 influxDB)。
下载 jmxtrans rpm 安装包
wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270.rpm
–2019-04-10 03:18:14– http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270.rpm
Resolving central.maven.org (central.maven.org)… 151.101.40.209
Connecting to central.maven.org (central.maven.org)|151.101.40.209|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 18750744 (18M) [application/x-rpm]
Saving to:‘jmxtrans-270.rpm’
100%[================================================================================================================================================================================>] 18,750,744 342KB/s in 43s
2019-04-10 03:18:59 (422 KB/s) –‘jmxtrans-270.rpm’saved [18750744/18750744]
安装 rpm 包
rpm -ivh jmxtrans-270.rpm
Preparing… ################################# [100%]
Updating / installing…
1:jmxtrans-270-1 ################################# [100%]
jmxtrans 相关路径
jmxtrans 安装目录:/usr/share/jmxtrans
json 文件默认目录:/var/lib/jmxtrans/
日志路径:/var/log/jmxtrans/jmxtrans.log
配置 json,jmxtrans 的 github 上有一段示例配置
{
“servers” : [{
“port” : “1099”,
“host” : “w2”,
“queries” : [{
“obj” : “java.lang:type=Memory”,
“attr” : [“HeapMemoryUsage”, “NonHeapMemoryUsage”],
“resultAlias”:”jvmMemory”,
“outputWriters” : [{
“@class” : “com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory”,
“url” : “http://127.0.0.1:8086/”,
“username” : “admin”,
“password” : “admin”,
“database” : “jmxDB”,
“tags” : {“application” : “kafka”}
} ]
} ]
} ]
}
host:监控服务器
port:jmx 端口
obj:对应 jmx 的 ObjectName,就是我们要监控的指标
attr:对应 ObjectName 的属性,可以理解为我们要监控的指标的值
resultAlias:对应 metric 的名称,在 InfluxDB 里面就是 MEASUREMENTS 名
tags:对应 InfluxDB 的 tag 功能,对与存储在同一个 MEASUREMENTS 里面的不同监控指标可以做区分,我们在用 Grafana 绘图的时候会用到,建议对每个监控指标都打上 tags
启动 jmxtrans
service jmxtrans start
Starting JmxTrans…
查看日志没有报错即为成功
tail /var/log/jmxtrans/jmxtrans.log
INFO | jvm 1 | 2019/04/10 04:44:31 | Using thread pool ‘org.quartz.simpl.SimpleThreadPool’ – with 10 threads.
INFO | jvm 1 | 2019/04/10 04:44:31 | Using job-store ‘org.quartz.simpl.RAMJobStore’ – which does not support persistence. and is not clustered.
INFO | jvm 1 | 2019/04/10 04:44:31 |
INFO | jvm 1 | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO org.quartz.impl.StdSchedulerFactory – Quartz scheduler ‘ServerScheduler’ initialized from an externally opened InputStream.
INFO | jvm 1 | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO org.quartz.impl.StdSchedulerFactory – Quartz scheduler version: 1.8.6
INFO | jvm 1 | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO org.quartz.core.QuartzScheduler – JobFactory set to: com.googlecode.jmxtrans.guice.GuiceJobFactory@23822296
2019-04-10 04:44:31 [WrapperSimpleAppMain] level com.googlecode.jmxtrans.JmxTransformer [JmxTransformer.java:177] – Starting Jmxtrans on : /var/lib/jmxtrans
2019-04-10 04:44:31 [WrapperSimpleAppMain] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:519] – Scheduler ServerScheduler_$_node11554885871753 started.
INFO | jvm 1 | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO c.googlecode.jmxtrans.JmxTransformer – Starting Jmxtrans on : /var/lib/jmxtrans
INFO | jvm 1 | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO org.quartz.core.QuartzScheduler – Scheduler ServerScheduler_$_node11554885871753 started.
附上两段通用的 json 文件
base_127.0.0.1.json
View Code
topicA_1.json
View Code
安装 Grafana
Grafana 是一个跨平台的开源的度量分析和可视化工具,可以通过将采集的数据查询然后可视化的展示,并及时通知。
下载 jmxtrans rpm 安装包
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-6.0.2-1.x86_64.rpm
–2019-04-10 04:53:15– https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-6.0.2-1.x86_64.rpm
Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)… 52.218.144.92
Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.144.92|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 56002012 (53M) [application/x-RedHat-package-manager]
Saving to:‘grafana-6.0.2-1.x86_64.rpm’
100%[================================================================================================================================================================================>] 56,002,012 177KB/s in 2m 52s
2019-04-10 04:56:08 (318 KB/s) –‘grafana-6.0.2-1.x86_64.rpm’saved [56002012/56002012]
安装 rpm 包
rpm -ivh grafana-6.0.2-1.x86_64.rpm
warning: grafana-6.0.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
error: Failed dependencies:
fontconfig is needed by grafana-6.0.2-1.x86_64
urw-fonts is needed by grafana-6.0.2-1.x86_64
缺少依赖,下载依赖
yum install –downloadonly –downloaddir=./ fontconfig
yum localinstall fontconfig-2.13.0-4.3.el7.x86_64.rpm
yum install –downloadonly –downloaddir=./ urw-fonts
yum localinstall urw-fonts-2.4-16.el7.noarch.rpm
rpm -ivh grafana-6.0.2-1.x86_64.rpm
warning: grafana-6.0.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
Preparing… ################################# [100%]
Updating / installing…
1:grafana-6.0.2-1 ################################# [100%]
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server.service
### You can start grafana-server by executing
sudo /bin/systemctl start grafana-server.service
POSTTRANS: Running script
启动 Grafana
service grafana-server start
Starting grafana-server (via systemctl): [OK]
打开浏览器
http://127.0.0.1:3000
先输入默认用户名密码 admin/admin
设置新密码
点击 Add data source
选择 InfluxDB
输入连接信息后点击 Save & Test
通过后点击 Back 返回
左侧 + 可以创建或引入仪表盘
类似于数据库 SQL 语句,查询相应的指标
计算平均每秒数值可以使用如上语法,用当前值减 1 分钟之前的值再除以 60
具体展示效果就看各位的审美能力,这里就不贴出来了。至此,Kafka 的 JMX 指标监控就完成了。
:
![post-qrcode](https://www.xgss.net/wp-content/uploads/2023/07/二维码-微信公众号256x256.jpg)