共计 20567 个字符，预计需要花费 52 分钟才能阅读完成。
HugePages 是集成到 Linux kernel 2.6 中的一个功能。启用 HugePages 可以使用操作系统来支持比缺省的内存页 (4KB) 更大的内存页。使用非常大的内存页大小可以通过减少访问页表条目所需要的系统资源数量而提高系统性能。HugePages 对于 32 位与 64 位系统都是有效的。HugePage 的大小范围从 2MB 到 256MB，依赖于内核版本和硬件架构。对于 Oracle 数据库，使用 HugePages 减少操作系统维护内存页
状态并增加 Translation Lookaside Buffer(TLB) 的撞击率。
1. 使用 HugePages 来优化 SGA
不使用 HugePages 时，操作系统将保持每个内存页大小为 4KB，当为 SGA 分配内存页时，操作系统内核必须对分配给 SGA 的每个 4KB 页使用页生命周期(脏，可用，映射到进程，等等）持续更新。
使用 HugePages 时，操作系统页表 (虚拟内存到物理内存的映射) 很小，因为每个页表条目指向的内存页大小从 2MB 到 256MB。同时内核有比较少的内存页生命周期被监控。例如，如果 64 位硬件使用 HugePages，并且想要映射 256MB 的内存，你可能只需要一个页表条目(PTE)。如果不使用 HugePages 并且想要映射 256MB 内存，那么必须有 256*1024KB/4KB=65536 个 PTEs。
HugePages 提供了以下优点:
通过增加 TLB 撞击率来提高性能
内存页被锁定在内存中并且不会发生交换，对共享内存结构比如 SGA 提供了随机访问
连续内存页预分配除了用于系统的共享内存比如 SGA 不能用于其它的目的
因为使用大的内存页大小所以虚拟内存相关的内核有较少性能开销
2 对 Linux 配置 HugePages
运行以下命令来判断内核是否支持 HugePages:
[root@jyrac1 ~]# uname -r
2.6.18-164.el5
[root@jyrac1 ~]# grep Huge /proc/meminfo
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
有一些 Linux 缺省情况下是不支持 HugePages 的。对于这样的系统使用 config_hugetlbfs 和 config_hugetlb_page 配置选项来构建 Linux 内核。config_hugetlbfs 位于文件系统并且当你选择 config_hugetlbfs 时需要同时选择 config_hugetlb_page。
编辑 /etc/security/limits.conf 文件来设置 memlock。memlock 设置以 KB 为单位，并且当启用 HugePages 内存时，最大锁定内存限制应该被设置为当前可随机访问内存的 90%，当没有启用 HugePages 内存时，最大锁定内存限制应该被设置成至少 3145728KB(3GB)。例如，如果有 2G 可随机访问内存，并且增加以下条目来增加最大锁定内存地址空间:
[root@jyrac1 ~]# vi /etc/security/limits.conf
grid soft memlock 2097152
grid hard memlock 2097152
oracle soft memlock 2097152
oracle hard memlock 2097152
也可以将 memlock 的值设置为比 SGA 的值大
以 grid 用户登录，并执行 ulimit - l 命令来验证新设置的 memlock 是否生效
[grid@jyrac1 ~]$ ulimit -l
2097152
以 oracle 用户登录，并执行 ulimit - l 命令来验证新设置的 memlock 是否生效
[oracle@jyrac1 ~]$ ulimit -l
2097152
运行以下命令来显示 Hugepagesize 变量:
[oracle@jyrac1 ~]$ grep Hugepagesize /proc/meminfo
Hugepagesize: 2048 kB
完成以下过程来创建一个脚本用来为当前共享内存段计算 hugepages 配置的建议值创建一个 hugepages_settings.sh 脚本并增加以下内容:
[root@jyrac1 /]# vi hugepages_settings.sh
#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
# on Oracle Linux
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support
# http://support.oracle.com
# Welcome text
echo “
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
* For ASM instance, it needs to configure ASMM instead of AMM.
* The ‘pga_aggregate_target’ is outside the SGA and
you should accommodate this while calculating SGA size.
* In case you changes the DB SGA size,
as the new SGA will not fit in the previous HugePages configuration,
it had better disable the whole HugePages,
start the DB with new SGA size and run the script again.
And make sure that:
* Oracle Database instance(s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
(See Doc ID 749851.1)
* The shared memory segments can be listed by command:
# ipcs -m
Press Enter to proceed…”
read
# Check for the kernel version
KERN=`uname -r | awk -F. ‘{printf(“%d.%d\n”,$1,$2); }’`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk ‘{print $2}’`
if [-z “$HPG_SZ”];then
echo “The hugepages may not be supported in the system where the script is being executed.”
exit 1
fi
# Initialize the counter
NUM_PG=0
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk ‘{print $1}’ | grep “[0-9][0-9]*”`
do
MIN_PG=`echo “$SEG_BYTES/($HPG_SZ*1024)” | bc -q`
if [$MIN_PG -gt 0]; then
NUM_PG=`echo “$NUM_PG+$MIN_PG+1” | bc -q`
fi
done
RES_BYTES=`echo “$NUM_PG * $HPG_SZ * 1024” | bc -q`
# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [$RES_BYTES -lt 100000000]; then
echo “***********”
echo “** ERROR **”
echo “***********”
echo “Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:
# ipcs -m
of a size that can match an Oracle Database SGA. Please make sure that:
* Oracle Database instance is up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not configured”
exit 1
fi
# Finish with results
case $KERN in
‘2.2’) echo “Kernel version $KERN is not supported. Exiting.” ;;
‘2.4’) HUGETLB_POOL=`echo “$NUM_PG*$HPG_SZ/1024” | bc -q`;
echo “Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL” ;;
‘2.6’) echo “Recommended setting: vm.nr_hugepages = $NUM_PG” ;;
‘3.8’) echo “Recommended setting: vm.nr_hugepages = $NUM_PG” ;;
‘3.10’) echo “Recommended setting: vm.nr_hugepages = $NUM_PG” ;;
‘4.1’) echo “Recommended setting: vm.nr_hugepages = $NUM_PG” ;;
esac
# End
执行以下命令来改变 hugepages_settings.sh 脚本的权限
[root@jyrac1 /]# chmod +x hugepages_settings.sh
运行 hugepages_settings.sh 脚本来计算 hugepages 配置的参数值
[root@jyrac1 /]# ./hugepages_settings.sh
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
* For ASM instance, it needs to configure ASMM instead of AMM.
* The ‘pga_aggregate_target’ is outside the SGA and
you should accommodate this while calculating SGA size.
* In case you changes the DB SGA size,
as the new SGA will not fit in the previous HugePages configuration,
it had better disable the whole HugePages,
start the DB with new SGA size and run the script again.
And make sure that:
* Oracle Database instance(s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
(See Doc ID 749851.1)
* The shared memory segments can be listed by command:
# ipcs -m
Press Enter to proceed…
***********
** ERROR **
***********
Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:
# ipcs -m
of a size that can match an Oracle Database SGA. Please make sure that:
* Oracle Database instance is up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not configured
从上面的信息可以看到需要确认 Oracle 实例是否正在运行，如果是 Oracle 11g 不能使用 AMM
[root@jyrac1 ~]# ps -ef | grep pmon
grid 4116 1 0 Apr18 ? 00:00:03 asm_pmon_+ASM1
oracle 4944 1 0 Apr18 ? 00:00:03 ora_pmon_jyrac1
root 18184 29273 0 15:15 pts/1 00:00:00 grep pmon
上面信息可以看到 Oracle 实例正在运行。
[grid@jyrac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Apr 20 15:20:23 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set long 900
SQL> set linesize 900
SQL> show parameter instance_name
NAME TYPE VALUE
———————————— ———————- ——————————
instance_name string +ASM1
SQL> show parameter memory
NAME TYPE VALUE
———————————— ———————- ——————————
memory_max_target big integer 1076M
memory_target big integer 1076M
[oracle@jyrac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Wed Apr 20 15:21:04 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> set long 900
SQL> set linesize 900
SQL> show parameter instance_name
NAME TYPE VALUE
———————————— ———– ——————————
instance_name string jyrac1
SQL> show parameter memory
NAME TYPE VALUE
———————————— ———– ——————————
hi_shared_memory_address integer 0
memory_max_target big integer 2G
memory_target big integer 2G
shared_memory_address integer 0
确实 asm 与数据库实例都启用了 AMM，需要禁用 AMM 但是可以使用 ASMM 修改 ASM 实例，禁用 AMM，但使用 ASMM, 如果是 RAC 所有节点都需要修改
SQL> alter system set sga_max_size=640M scope=spfile sid=’*’;
System altered.
SQL> alter system set sga_target=640M scope=spfile sid=’*’;
System altered.
SQL> alter system set pga_aggregate_target=320M scope=spfile sid=’*’;
System altered.
SQL> alter system set memory_target=0 scope=spfile sid=’*’;
System altered.
这里对于 memory_target 不能使用 reset 否则会出现以下错误:
SQL> startup
ORA-01078: failure in processing system parameters
ORA-00843: Parameter not taking MEMORY_MAX_TARGET into account
ORA-00849: SGA_TARGET 671088640 cannot be set to more than MEMORY_MAX_TARGET 0.
SQL> alter system reset memory_max_target scope=spfile sid=’*’;
System altered.
修改数据库实例，禁用 AMM，但使用 ASMM, 如果是 RAC 所有节点都需要修改
SQL> alter system set sga_max_size=640M scope=spfile sid=’*’;
System altered.
SQL> alter system set sga_target=640M scope=spfile sid=’*’;
System altered.
SQL> alter system set pga_aggregate_target=320M scope=spfile sid=’*’;
System altered.
SQL> alter system reset memory_max_target scope=spfile sid=’*’;
System altered.
SQL> alter system reset memory_target scope=spfile sid=’*’;
System altered.
重启 ASM 与数据库实例, 如果是 RAC 所有节点都需要重启，首先停止 ASM 与数据库实例
[grid@jyrac1 ~]$ srvctl stop asm -n jyrac1 -f
[grid@jyrac1 ~]$ srvctl stop asm -n jyrac2 -f
[grid@jyrac1 ~]$ srvctl stop database -d jyrac
[grid@jyrac1 ~]$ crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.CRSDG.dg
OFFLINE OFFLINE jyrac1
OFFLINE OFFLINE jyrac2
ora.DATADG.dg
OFFLINE OFFLINE jyrac1
OFFLINE OFFLINE jyrac2
ora.LISTENER.lsnr
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.asm
OFFLINE OFFLINE jyrac1 Instance Shutdown
OFFLINE OFFLINE jyrac2 Instance Shutdown
ora.gsd
ONLINE OFFLINE jyrac1
ONLINE OFFLINE jyrac2
ora.net1.network
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.ons
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.registry.acfs
OFFLINE OFFLINE jyrac1
OFFLINE OFFLINE jyrac2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE jyrac2
ora.cvu
1 ONLINE ONLINE jyrac2
ora.jyrac.db
1 OFFLINE OFFLINE Instance Shutdown
2 OFFLINE OFFLINE Instance Shutdown
ora.jyrac1.vip
1 ONLINE ONLINE jyrac1
ora.jyrac2.vip
1 ONLINE ONLINE jyrac2
ora.oc4j
1 ONLINE ONLINE jyrac2
ora.scan1.vip
1 ONLINE ONLINE jyrac2
启动 ASM 与数据库实例
grid@jyrac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Apr 20 17:48:32 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ASM instance started
Total System Global Area 669581312 bytes
Fixed Size 1366724 bytes
Variable Size 643048764 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL> show parameter instance_name
NAME TYPE VALUE
———————————— ———————- ——————————
instance_name string +ASM2
SQL> show parameter memory
NAME TYPE VALUE
———————————— ———————- ——————————
memory_max_target big integer 0
memory_target big integer 0
SQL> show parameter sga
NAME TYPE VALUE
———————————— ———————- ——————————
lock_sga boolean FALSE
sga_max_size big integer 640M
sga_target big integer 640M
grid@jyrac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Apr 20 17:48:32 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ASM instance started
Total System Global Area 669581312 bytes
Fixed Size 1366724 bytes
Variable Size 643048764 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL> show parameter instance_name
NAME TYPE VALUE
———————————— ———————- ——————————
instance_name string +ASM2
SQL> show parameter memory
NAME TYPE VALUE
———————————— ———————- ——————————
memory_max_target big integer 0
memory_target big integer 0
SQL> show parameter sga
NAME TYPE VALUE
———————————— ———————- ——————————
lock_sga boolean FALSE
sga_max_size big integer 640M
sga_target big integer 640M
[grid@jyrac1 ~]$ crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.CRSDG.dg
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.DATADG.dg
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.LISTENER.lsnr
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.asm
ONLINE ONLINE jyrac1 Started
ONLINE ONLINE jyrac2 Started
ora.gsd
ONLINE OFFLINE jyrac1
ONLINE OFFLINE jyrac2
ora.net1.network
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.ons
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.registry.acfs
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE jyrac1
ora.cvu
1 ONLINE ONLINE jyrac1
ora.jyrac.db
1 OFFLINE OFFLINE Instance Shutdown
2 OFFLINE OFFLINE Instance Shutdown
ora.jyrac1.vip
1 ONLINE ONLINE jyrac1
ora.jyrac2.vip
1 ONLINE ONLINE jyrac2
ora.oc4j
1 ONLINE ONLINE jyrac1
ora.scan1.vip
1 ONLINE ONLINE jyrac1
从上面的信息可以看到 asm 实例已经启动了并且禁用了 AMM
[grid@jyrac1 ~]$ srvctl start database -d jyrac
[grid@jyrac1 ~]$ crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.CRSDG.dg
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.DATADG.dg
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.LISTENER.lsnr
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.asm
ONLINE ONLINE jyrac1 Started
ONLINE ONLINE jyrac2 Started
ora.gsd
ONLINE OFFLINE jyrac1
ONLINE OFFLINE jyrac2
ora.net1.network
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.ons
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
ora.registry.acfs
ONLINE ONLINE jyrac1
ONLINE ONLINE jyrac2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE jyrac1
ora.cvu
1 ONLINE ONLINE jyrac1
ora.jyrac.db
1 ONLINE ONLINE jyrac1 Open
2 ONLINE ONLINE jyrac2 Open
ora.jyrac1.vip
1 ONLINE ONLINE jyrac1
ora.jyrac2.vip
1 ONLINE ONLINE jyrac2
ora.oc4j
1 ONLINE ONLINE jyrac1
ora.scan1.vip
1 ONLINE ONLINE jyrac1
SQL> show parameter instance_name
NAME TYPE VALUE
———————————— ———– ——————————
instance_name string jyrac1
SQL> show parameter memory
NAME TYPE VALUE
———————————— ———– ——————————
hi_shared_memory_address integer 0
memory_max_target big integer 0
memory_target big integer 0
shared_memory_address integer 0
SQL> show parameter sga
NAME TYPE VALUE
———————————— ———– ——————————
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 640M
sga_target big integer 640M
SQL> show parameter instance_name
NAME TYPE VALUE
———————————— ———– ——————————
instance_name string jyrac2
SQL> show parameter memory
NAME TYPE VALUE
———————————— ———– ——————————
hi_shared_memory_address integer 0
memory_max_target big integer 0
memory_target big integer 0
shared_memory_address integer 0
SQL> show parameter sga
NAME TYPE VALUE
———————————— ———– ——————————
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 640M
sga_target big integer 640M
数据库也已经成功启动并且禁用了 AMM
再次执行 hugepages_settings.sh 脚本计算 HugePages 的大小
[root@jyrac1 /]# ./hugepages_settings.sh
Recommended setting: vm.nr_hugepages = 649
编辑 /etc/sysctl.conf 文件增加参数 vm.nr_hugepages = 649, 并执行 sysctl - p 命令使用修改立即生效，但 oracle 实例并没有使用 HugePages 从 HugePages_Total 与 HugePages_Free 相等可以判断出来。
[root@jyrac1 /]# vi /etc/sysctl.conf
vm.nr_hugepages = 649
[root@jyrac1 /]# sysctl -p
[root@jyrac1 /]# grep Huge /proc/meminfo
HugePages_Total: 649
HugePages_Free: 649
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
重启实例
SQL> startup
ASM instance started
Total System Global Area 669581312 bytes
Fixed Size 1366724 bytes
Variable Size 643048764 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
查看 asm 实例的 alert_+ASM1.log 可以看到如下信息:
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 2048 MB
Total Shared Global Region in Large Pages = 642 MB (100%)
Large Pages used by this instance: 321 (642 MB)
Large Pages unused system wide = 328 (656 MB)
Large Pages configured system wide = 649 (1298 MB)
Large Page size = 2048 KB
SQL> startup
ORACLE instance started.
Total System Global Area 669581312 bytes
Fixed Size 1366724 bytes
Variable Size 243270972 bytes
Database Buffers 419430400 bytes
Redo Buffers 5513216 bytes
Database mounted.
Database opened.
查看实例 jyrac1 的 alert_jyrac1.log 可以看到如下信息:
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 2048 MB
Total Shared Global Region in Large Pages = 642 MB (100%)
Large Pages used by this instance: 321 (642 MB)
Large Pages unused system wide = 7 (14 MB)
Large Pages configured system wide = 649 (1298 MB)
Large Page size = 2048 KB
[root@jyrac1 /]# grep Huge /proc/meminfo
HugePages_Total: 649
HugePages_Free: 239
HugePages_Rsvd: 232
Hugepagesize: 2048 kB
从上面的信息可以看到已经使用了 Hugepages
3.HugePages 的限制
HugePages 有以下限制:
a. 对于 Oracle 11g 及以上版本数据库实例必须对 memory_target 与 memory_max_target 参数执行 alter system reset 命令，但对于 ASM 实例，对于 memory_target 参数只能设置为 0。
b.AMM 与 HugePages 是不兼容的，当使用 AMM，整个 SGA 内存通过在 /dev/shm 创建文件来进行内存的分配, 当使用 AMM 分配 SGA 时，HugePages 不会被保留。
c. 如果在 32 位系统中使用 VLM，那么对数据库 buffer cache 不能使用 HugePages。但对于 SGA 中的其它组件比如 shared_pool,
large_pool 等等可以使用 HugePages。对于 VLM(buffer cache) 分配内存是通过使用共享内存文件系统（ramfs/tmpfs/shmfs)来实现的。
d.HugePgaes 在系统启动后不受分配或释放，除非系统管理员通过修改可用页数或改变池大小来改变 HugePages 的配置。如果在系统启动时内存中没有保留所需要内存空间，那么 HugePages 会分配失败。
e. 确保 HugePages 配置合理，如果内存耗尽，应用将不能使用 HugePages。
f. 如果当实例启动用没有足够的 HugePages 并且参数 use_large_pages 设置为 only, 那么 Oracle 数据库将会启动失败并向 alert.log 中记录相关信息。
更多 Oracle 相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=12
本文永久更新链接地址：http://www.linuxidc.com/Linux/2016-04/130543.htm
