共计 11118 个字符，预计需要花费 28 分钟才能阅读完成。
information_schema 有何用？
- 通过 I_S 获取 MySQL 的一些元数据信息
- 获取表的数据文件、索引文件的大小、碎片情况、表行数、自增列增长情况等
- 获取正在运行的事务有那些，是否有阻塞等
- 获取当前 mysql 的连接 processlist 等等
mysql8.0 之前的查询方式
会在查询 information_schema 某个表时创建临时表
- 来自文件的元数据，扫描文件系统获取 FRM 文件的表定义
- 存储引擎的详细信息，例如动态表统计信息
- 来自 MySQL 服务器中全局数据结构的数据
在表的数量很多时，每次查询 I_S 会从文件系统中读取每个单独的 FRM 文件，使用更多的 CPU 周期来打开表并准备相关的内存数据结构
mysql8.0 开始的查询方式
- 引入了基于 InnoDB 的本地数据字典表
- 表中不在有 FRM 表定义文件
- 所有数据库表的元数据都存储在事务数据字典表中
- I_S 中表被设计为数据字典表上的 VIEW（有些还是临时表 0_0）
消除了以下成本
查询 INFORMATION_SCHEMA 时创建的临时表
扫描文件系统目录以查找 FRM 文件
改进
利用 MySQL 优化器的全部功能，使用数据字典表上的索引来更好的查询
mysql5.7 中表文件
ll test*
Jul 10 10:52 testse.frm
Jul 10 10:52 testse.ibd
mysql8.0 中表文件
ll test*
Jul 10 10:25 testse.ibd
mysql5.7 和 mysql8.0 I_S 中 tables 表的存在方式
| mysql5.7.22 | |
| show create table information_schema.tables\G | |
| *************************** 1. row *************************** | |
| Table: TABLES | |
| Create Table: CREATE TEMPORARY TABLE `TABLES` (`TABLE_CATALOG` varchar(512) NOT NULL DEFAULT '', | |
| `TABLE_SCHEMA` varchar(64) NOT NULL DEFAULT '', | |
| `TABLE_NAME` varchar(64) NOT NULL DEFAULT '', | |
| `TABLE_TYPE` varchar(64) NOT NULL DEFAULT '', | |
| `ENGINE` varchar(64) DEFAULT NULL, | |
| mysql8.0.15 | |
| show create table information_schema.tables\G | |
| *************************** 1. row *************************** | |
| View: TABLES | |
| Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`mysql.infoschema`@`localhost` SQL SECURITY DEFINER VIEW ` |
- mysql5.7 中 I_S 中 tables 表是以临时表的形式存在的（查询该表就会创建临时表，创建的临时表过多，可能会导致 mysql 占用的内存暴涨，出现 OOM）
- mysql8.0 中 I_S 中 tables 表以视图的形式存在（查询该视图，不会创建临时表，会使用到视图中表的索引）
mysql5.7 中获取表大小情况
| SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT(FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse'; | |
| +------------+-------------+--------------+------------+----------------+------------+---------------------+ | |
| | table_name | dbdata_size | dbindex_size | db_size(G) | AVG_ROW_LENGTH | table_rows | update_time | | |
| +------------+-------------+--------------+------------+----------------+------------+---------------------+ | |
| | testse | 0.02M | 0.02M | 0.00G | 862 | 19 | 2019-07-10 10:52:02 | | |
| +------------+-------------+--------------+------------+----------------+------------+---------------------+ | |
| 执行计划中出现了 where，未用到索引（存储引擎检索数据后，server 层进行过滤）desc SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT(FORMAT(data_length + index_length / 1024 / 1024 / 1024, 2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse'; | |
| +----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+ | |
| | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | | |
| +----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+ | |
| | 1 | SIMPLE | tables | NULL | ALL | NULL | TABLE_SCHEMA,TABLE_NAME | NULL | NULL | NULL | NULL | Using where; Open_full_table; Scanned 0 databases | | |
| +----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+ | |
mysql8.0 中获取表大小情况
| SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT(FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse'; | |
| +------------+-------------+--------------+------------+----------------+------------+---------------------+ | |
| | TABLE_NAME | dbdata_size | dbindex_size | db_size(G) | AVG_ROW_LENGTH | TABLE_ROWS | UPDATE_TIME | | |
| +------------+-------------+--------------+------------+----------------+------------+---------------------+ | |
| | testse | 0.02M | 0.02M | 0.00G | 862 | 19 | 2019-07-10 10:25:16 | | |
| +------------+-------------+--------------+------------+----------------+------------+---------------------+ | |
| 能使用到数据字典表的索引 | |
| desc SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT(FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse'; | |
| +----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+ | |
| | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | | |
| +----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+ | |
| | 1 | SIMPLE | cat | NULL | index | PRIMARY | name | 194 | NULL | 1 | 100.00 | Using index | | |
| | 1 | SIMPLE | sch | NULL | eq_ref | PRIMARY,catalog_id | catalog_id | 202 | mysql.cat.id,const | 1 | 100.00 | Using index | | |
| | 1 | SIMPLE | tbl | NULL | eq_ref | schema_id | schema_id | 202 | mysql.sch.id,const | 1 | 100.00 | Using where | | |
| | 1 | SIMPLE | stat | NULL | const | PRIMARY | PRIMARY | 388 | const,const | 1 | 100.00 | NULL | | |
| | 1 | SIMPLE | ts | NULL | eq_ref | PRIMARY | PRIMARY | 8 | mysql.tbl.tablespace_id | 1 | 100.00 | NULL | | |
| | 1 | SIMPLE | col | NULL | eq_ref | PRIMARY | PRIMARY | 8 | mysql.tbl.collation_id | 1 | 100.00 | Using index | | |
| +----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+ | |
测试 5.7 和 8.0 不同版本访问 I_S 库的性能
机器
| cat /etc/RedHat-release | xargs echo '版本' && dmidecode -s system-product-name | xargs echo '是否虚拟化' && cat /proc/cpuinfo |grep "processor"|wc -l | xargs echo 'cpu 核数' | |
| 版本 CentOS Linux release 7.5.1804 (Core) | |
| 是否虚拟化 KVM | |
| cpu 核数 4 |
1、分别在 mysql5.7 和 mysql8.0 中创建 5 个库，每个库中 30 个表（共 1500 个表），每个表记录数为 10000
| user=admin | |
| passwd=admin | |
| port=57222 | |
| host=127.0.0.1 | |
| #创建 5 个库, 分别在每个库中创建 30 个表 | |
| for i in {1..5};do | |
| mysql -u$user -p$passwd -P$port -h$host<<EOF | |
| create database if not exists test_${i}; | |
| EOF | |
| sysbench /usr/local/share/sysbench/oltp_read_write.lua --mysql_storage_engine=innodb --table-size=10000 --tables=30 --mysql-db=test_${i} --mysql-user=$user --mysql-password=$passwd --mysql-port=$port --mysql-host=$host --threads=8 --time=10 --report-interval=1 --events=0 --db-driver=mysql prepare | |
| done; |
2、自定义访问 I_S.tables 表的 sysbench 脚本
cat tests/mytest.lua
| require("oltp_common") | |
| function thread_init(thread_id) | |
| drv=sysbench.sql.driver() | |
| con=drv:connect() | |
| end | |
| local function get_rand_db() | |
| return sysbench.rand.uniform(1, 5) | |
| end | |
| function event(thread_id) | |
| local vid1 | |
| local dbprefix | |
| vid1=get_rand_db() | |
| dbprefix = "test_" | |
| -- 生成 5 个 db 中的随机一个 db, 如 test_1 ,test_5 | |
| -- vid2=string.format("'%s%s'",test_,vid1) | |
| vid2="'".. dbprefix .. vid1 .."'" | |
| con:query("SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2),'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2),'M') AS dbindex_size, CONCAT(FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2),'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema =" .. vid2) | |
| end | |
| function thread_done() | |
| con:disconnect() | |
| end |
3、脚本通过 sysbench 测试访问 I_S.tables 的 qps
cat test_I_S_access.sh
| user=admin | |
| passwd=admin | |
| host=127.0.0.1 | |
| #输入 mysql 端口参数 | |
| port=$1 | |
| # port=8015 | |
| #输入 sysbench 测试时间参数多少秒 | |
| run_time=$2 | |
| function get_create_tmp_tables(){ | |
| mysql -u$user -p$passwd -P$port -h$host<<EOF | |
| select @@version as 'mysql version'; | |
| show global status like '%tmp_table%'; | |
| EOF | |
| } 2>/dev/null | |
| function begin_test(){ | |
| cd /usr/local/share/sysbench | |
| sysbench ./tests/mytest.lua --mysql-db=test_1 --mysql-host=$host --mysql-port=$port --mysql-user=$user --mysql-password=$passwd --threads=40 --report-interval=10 --rand-type=uniform --time=$run_time run | |
| } | |
| service mysqld${port} restart | |
| get_create_tmp_tables | |
| begin_test | |
| get_create_tmp_tables |
4、mysql5.7 和 mysql8.0 主要配置
| mysql5.7.22 | |
| 5.7.22-log | |
| innodb_buffer_pool_size 128M | |
| innodb_log_buffer_size 64M | |
| innodb_log_file_size 48M | |
| binlog_format ROW | |
| innodb_flush_log_at_trx_commit 1 | |
| sync_binlog 1 | |
| mysql8.0.15 | |
| 8.0.15 | |
| innodb_buffer_pool_size 128M | |
| innodb_log_buffer_size 64M | |
| innodb_log_file_size 48M | |
| binlog_format ROW | |
| innodb_flush_log_at_trx_commit 1 | |
| sync_binlog 1 |
5、mysql5.7 开启测试
bash test_I_S_acess.sh 57222 60
| Shutting down MySQL.. SUCCESS! | |
| Starting MySQL.. SUCCESS! | |
| mysql version | |
| 5.7.22-log | |
| Variable_name Value | |
| Created_tmp_tables 1 | |
| sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3) | |
| Running the test with following options: | |
| Number of threads: 40 | |
| Report intermediate results every 10 second(s) | |
| Initializing random number generator from current time | |
| Initializing worker threads... | |
| Threads started! | |
| [10s] thds: 40 tps: 4765.73 qps: 4765.73 (r/w/o: 4765.73/0.00/0.00) lat (ms,95%): 18.95 err/s: 0.00 reconn/s: 0.00 | |
| [20s] thds: 40 tps: 5409.00 qps: 5409.00 (r/w/o: 5409.00/0.00/0.00) lat (ms,95%): 17.95 err/s: 0.00 reconn/s: 0.00 | |
| [30s] thds: 40 tps: 5154.45 qps: 5154.45 (r/w/o: 5154.45/0.00/0.00) lat (ms,95%): 18.61 err/s: 0.00 reconn/s: 0.00 | |
| [40s] thds: 40 tps: 5383.50 qps: 5383.50 (r/w/o: 5383.50/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s: 0.00 | |
| [50s] thds: 40 tps: 5456.11 qps: 5456.11 (r/w/o: 5456.11/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s: 0.00 | |
| [60s] thds: 40 tps: 5458.66 qps: 5458.66 (r/w/o: 5458.66/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s: 0.00 | |
| SQL statistics: | |
| queries performed: | |
| read: 316322 | |
| write: 0 | |
| other: 0 | |
| total: 316322 | |
| transactions: 316322 (5270.99 per sec.) | |
| queries: 316322 (5270.99 per sec.) | |
| ignored errors: 0 (0.00 per sec.) | |
| reconnects: 0 (0.00 per sec.) | |
| Throughput: | |
| events/s (eps): 5270.9892 | |
| time elapsed: 60.0119s | |
| total number of events: 316322 | |
| Latency (ms): | |
| min: 0.42 | |
| avg: 7.59 | |
| max: 858.18 | |
| 95th percentile: 17.95 | |
| sum: 2399738.58 | |
| Threads fairness: | |
| events (avg/stddev): 7908.0500/53.41 | |
| execution time (avg/stddev): 59.9935/0.00 | |
| mysql version | |
| 5.7.22-log | |
| Variable_name Value | |
| Created_tmp_tables 316327 |
cpu 使用率接近 100%
- mysql5.7.22 40 个并发线程 压力测试 60 秒，访问 I_S.tables 平均 qps 为 5k，并创建了 31w 个临时表!
6、mysql8.0 开启测试
| Shutting down MySQL.. SUCCESS! | |
| Starting MySQL.... SUCCESS! | |
| mysql version | |
| 8.0.15 | |
| Variable_name Value | |
| Created_tmp_tables 1 | |
| sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3) | |
| Running the test with following options: | |
| Number of threads: 40 | |
| Report intermediate results every 10 second(s) | |
| Initializing random number generator from current time | |
| Initializing worker threads... | |
| Threads started! | |
| [10s] thds: 40 tps: 1283.76 qps: 1283.76 (r/w/o: 1283.76/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00 | |
| [20s] thds: 40 tps: 1286.43 qps: 1286.43 (r/w/o: 1286.43/0.00/0.00) lat (ms,95%): 39.65 err/s: 0.00 reconn/s: 0.00 | |
| [30s] thds: 40 tps: 1333.00 qps: 1333.00 (r/w/o: 1333.00/0.00/0.00) lat (ms,95%): 37.56 err/s: 0.00 reconn/s: 0.00 | |
| [40s] thds: 40 tps: 1280.79 qps: 1280.79 (r/w/o: 1280.79/0.00/0.00) lat (ms,95%): 40.37 err/s: 0.00 reconn/s: 0.00 | |
| [50s] thds: 40 tps: 1313.57 qps: 1313.57 (r/w/o: 1313.57/0.00/0.00) lat (ms,95%): 38.94 err/s: 0.00 reconn/s: 0.00 | |
| [60s] thds: 40 tps: 1222.95 qps: 1222.95 (r/w/o: 1222.95/0.00/0.00) lat (ms,95%): 42.61 err/s: 0.00 reconn/s: 0.00 | |
| SQL statistics: | |
| queries performed: | |
| read: 77248 | |
| write: 0 | |
| other: 0 | |
| total: 77248 | |
| transactions: 77248 (1286.95 per sec.) | |
| queries: 77248 (1286.95 per sec.) | |
| ignored errors: 0 (0.00 per sec.) | |
| reconnects: 0 (0.00 per sec.) | |
| Throughput: | |
| events/s (eps): 1286.9486 | |
| time elapsed: 60.0242s | |
| total number of events: 77248 | |
| Latency (ms): | |
| min: 1.88 | |
| avg: 31.08 | |
| max: 692.98 | |
| 95th percentile: 40.37 | |
| sum: 2400505.33 | |
| Threads fairness: | |
| events (avg/stddev): 1931.2000/51.17 | |
| execution time (avg/stddev): 60.0126/0.00 | |
| mysql version | |
| 8.0.15 | |
| Variable_name Value | |
| Created_tmp_tables 2 |
cpu 使用率接近 100%
- mysql8.0.15 40 个并发线程，压力测试 60 秒，访问 I_S.tables 平均 qps 为 1.2k，并创建了 1 个临时表!
结论
- mysql8.0 开始查询 I_S 中表不会再从文件系统 (FRM, TRG，OPT 文件) 和 myisam、innodb 系统表读取元数据信息（8.0 开始元数据信息统一存放在数据字典表中）
- mysql8.0 访问 I_S.tables 不会创建临时表，这减少了内存暴涨的可能，但访问 I_S.tables 的 qps 大约是 mysql5.7.22 的1/5，访问速度没有 mysql5.7.22 的快
- mysql8.0 访问 I_S.tables 用户空间 cpu 消耗没有 mysql5.7 的高（没有创建临时表的缘故吧），但系统空间 cpu 消耗高于 mysql5.7！
：
正文完
星哥玩云-微信公众号
