共计 14069 个字符，预计需要花费 36 分钟才能阅读完成。
实验环境准备：
RHEL 6.5 + Oracle 11.2.0.4 RAC (2nodes)
- OCR 和 Voting Disk 使用的是 OCR1 磁盘组，底层对应 3 个 1G 大小的共享 LUN，一般冗余；
- DATA 使用的是 DATA1 磁盘组，底层对应的是 3 个 5G 大小的共享 LUN，外部冗余；
- FRA 使用的是 FRA1 磁盘组，底层对应的是 1 个 5G 大小的共享 LUN，外部冗余；
- 额外添加 7 个共享 LUN(3 个 1G 大小，4 个 5G 大小)，用来模拟新的存储映射出来的共享 LUN；
我这里是使用 Oracle VM VirtualBox 4.3 虚拟机软件模拟这个环境，
默认初始的 RAC 环境已经安装完毕，且已经存有业务数据。
对于 RAC 环境的安装可参考：
- Linux 平台 Oracle 11gR2 RAC 安装 http://www.linuxidc.com/Linux/2015-09/122612.htm
模拟的业务数据可参考 模拟业务最小测试用例 第 1～5 步进行初始化。
初始的 RAC 环境配置信息：
#public ip
192.168.56.150 linuxidc
192.168.56.152 jyrac2
#private ip
10.10.10.11 linuxidc-priv
10.10.10.12 jyrac2-priv
#virtual ip
192.168.56.151 linuxidc-vip
192.168.56.153 jyrac2-vip
#scan ip
192.168.56.160 jyrac-scan本次需求：
更换 RAC 的共享存储，即把之前的存储设备划出的共享 LUN 上的所有数据全部迁移新的存储设备划分的 LUN 上。
具体就涉及到 OCR，voting disk 迁移，DATA 迁移。
在我这里实验具体就是指 OCR1，DATA1，FRA1 磁盘组的迁移。
一、准备工作
- 1.1 查看 asm 磁盘组
- 1.2 查看 asm 的磁盘信息
- 1.3 查看存储设备信息，确定新的存储
二、更换存储
- 2.1 迁移 OCR1 到 OCR2
- 2.2 迁移 DATA1
- 2.3 迁移 FRA1
- 2.4 测试
一、准备工作
1.1 查看 asm 磁盘组
select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup;
SQL> select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup;
NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.01.2 查看 asm 的磁盘信息
select GROUP_NUMBER, DISK_NUMBER, TOTAL_MB, FREE_MB, NAME, PATH from v$asm_disk order by 1,2;
SQL> set linesize 1000
SQL> r
1* select GROUP_NUMBER, DISK_NUMBER, TOTAL_MB, FREE_MB, NAME, PATH from v$asm_disk order by 1,2
GROUP_NUMBER DISK_NUMBER TOTAL_MB FREE_MB NAME PATH
1 0 1024 714 OCR1_0000 /dev/asm-diskb
1 1 1024 716 OCR1_0001 /dev/asm-diskc
1 2 1024 716 OCR1_0002 /dev/asm-diskd
2 0 5120 4552 DATA1_0000 /dev/asm-diske
2 1 5120 4549 DATA1_0001 /dev/asm-diskf
2 2 5120 4549 DATA1_0002 /dev/asm-diskg
3 0 5120 4790 FRA1_0000 /dev/asm-diskh
7 rows selected.1.3 查看存储设备信息，确定新的存储
这里实验环境是使用 virtualbox 模拟添加 7 个大小和之前一样的共享存储；
即分别为 1G 大小的 3 个（OCR2），5G 大小的 3 个（DATA1），5G 大小的 1 个（FRA1）。
NewStorage1
NewStorage2
NewStorage3
NewStorage4
NewStorage5
NewStorage6
NewStorage7
在系统中就是 /dev/sdi j k l m n o
udev 绑定（两个节点 root 执行）
--New Shared Disks
for i in i j k l m n o;
do
echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
doneudev 绑定后对应就是 asm-diski,j,k,l,m,n,o
start_udev（两个节点 root 执行）
查看某个磁盘大小：
[root@linuxidc ~]# fdisk -l /dev/asm-diski
Disk /dev/asm-diski: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000确定 /dev/asm-diski，/dev/asm-diskj，/dev/asm-diskk 对应的是新加的 1G 大小的 3 个盘
用于创建新的 OCR2
二、更换存储
2.1 迁移 OCR1 到 OCR2
2.1.1 创建新的 OCR2 磁盘组
su – grid
sqlplus / as sysasm
CREATE DISKGROUP OCR2 NORMAL REDUNDANCY DISK ‘/dev/asm-diski’, ‘/dev/asm-diskj’, ‘/dev/asm-diskk’ ATTRIBUTE ‘compatible.asm’=’11.2’;
SQL> CREATE DISKGROUP OCR2 NORMAL REDUNDANCY DISK '/dev/asm-diski', '/dev/asm-diskj', '/dev/asm-diskk' ATTRIBUTE 'compatible.asm'='11.2';
Diskgroup created.查看两个节点是否添加了 OCR2：
-- 节点1：SQL> r
1* select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup
NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
------------------------------------------------------------ ---------- ---------- ---------------------- ------------ ------------------------------------------------------------------------------------------------------------------------
DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
OCR2 3072 2913 MOUNTED NORMAL 11.2.0.0.0
-- 节点2：SQL> select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup
2 ;
NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
------------------------------------------------------------ ---------- ---------- ---------------------- ------------ ------------------------------------------------------------------------------------------------------------------------
DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
OCR2 0 0 DISMOUNTED 0.0.0.0.0发现节点 2 需要手动 mount 新加的 ASM 磁盘组
sqlplus / as sysasm
SQL> alter diskgroup OCR2 mount;
Diskgroup altered.2.1.2 添加 OCR 信息到 OCR2
more /etc/oracle/ocr.loc
ocrconfig -add +OCR2
ocrcheck -config
more /etc/oracle/ocr.loc
[root@linuxidc ~]# more /etc/oracle/ocr.loc
ocrconfig_loc=+OCR1
local_only=FALSE
[root@linuxidc ~]# ocrconfig -add +OCR2
[root@linuxidc ~]#
[root@linuxidc ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
Device/File Name : +OCR1
Device/File Name : +OCR2
[root@linuxidc ~]# more /etc/oracle/ocr.loc
#Device/file getting replaced by device +OCR2
ocrconfig_loc=+OCR1
ocrmirrorconfig_loc=+OCR2
local_only=false
[root@linuxidc ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3116
Available space (kbytes) : 259004
ID : 1718040627
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File Name : +OCR2
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE ab583d47c34a4f0ebfd46c8b04d7332b (/dev/asm-diskb) [OCR1]
2. ONLINE 853e95652d994fe3bfdf21ca23e65dfc (/dev/asm-diskc) [OCR1]
3. ONLINE 9f712a096a7b4f9bbfd811bff6462157 (/dev/asm-diskd) [OCR1]
Located 3 voting disk(s).2.1.3 迁移 VOTE DISK 到新创建的 ASM 磁盘组 OCR2 上
crsctl replace votedisk +OCR2（一个节点 root 执行）
先查询：
SQL> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB VO COMPATIBILITY
------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ---------- -- ------------------------------------------------------------------------------------------------------------------------
1 DATA1 MOUNTED EXTERN 15360 13502 N 11.2.0.0.0
2 FRA1 MOUNTED EXTERN 5120 4790 N 11.2.0.0.0
3 OCR1 MOUNTED NORMAL 3072 2146 Y 11.2.0.0.0
4 OCR2 MOUNTED NORMAL 3072 2244 N 11.2.0.0.0替换为新添加的 OCR2：
[root@linuxidc ~]# crsctl replace votedisk +OCR2
Successful addition of voting disk 83f3dea9f4c24fccbf05eaa163c5c087.
Successful addition of voting disk ad21c1f0452e4f4abff906127ff9fcc3.
Successful addition of voting disk 2771819c65984f85bf366e74ee85781a.
Successful deletion of voting disk ab583d47c34a4f0ebfd46c8b04d7332b.
Successful deletion of voting disk 853e95652d994fe3bfdf21ca23e65dfc.
Successful deletion of voting disk 9f712a096a7b4f9bbfd811bff6462157.
Successfully replaced voting disk group with +OCR2.
CRS-4266: Voting file(s) successfully replaced再查询：
SQL> r
1* select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB VO COMPATIBILITY
------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ---------- -- ------------------------------------------------------------------------------------------------------------------------
1 DATA1 MOUNTED EXTERN 15360 13502 N 11.2.0.0.0
2 FRA1 MOUNTED EXTERN 5120 4790 N 11.2.0.0.0
3 OCR1 MOUNTED NORMAL 3072 2242 N 11.2.0.0.0
4 OCR2 MOUNTED NORMAL 3072 2148 Y 11.2.0.0.0发现已经 voting file 已经在 OCR2 上，然后在两个节点 root 查询：
ocrcheck && crsctl query css votedisk
[root@linuxidc ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3116
Available space (kbytes) : 259004
ID : 1718040627
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File Name : +OCR2
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
Located 3 voting disk(s).
[root@jyrac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3116
Available space (kbytes) : 259004
ID : 1718040627
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File Name : +OCR2
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
Located 3 voting disk(s).2.1.4 创建 ASM 实例 spfile 到新创建的 OCR2 ASM 磁盘组上
一个节点 grid 用户登录 ASM 实例执行：
create pfile=’/tmp/asmpfile.ora’ from spfile;
create spfile=’+OCR2′ from pfile=’/tmp/asmpfile.ora’;
SQL> show parameter pfile
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
spfile string +OCR1/jyrac-scan/asmparameterf
ile/registry.253.919993739
SQL> create pfile='/tmp/asmpfile.ora' from spfile;
File created.
SQL> create spfile='+OCR2' from pfile='/tmp/asmpfile.ora';
File created.
SQL> show parameter pfile
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
spfile string +OCR1/jyrac-scan/asmparameterf
ile/registry.253.9199937392.1.5 删除 OCR 原位置 ASM 磁盘组 OCR1
一个节点 root 执行：
[root@linuxidc ~]# ocrconfig -delete +OCR1
两个节点 root 执行：
查看 OCR 和 VOTE 新状态与位置
[root@linuxidc ~]# ocrcheck && crsctl query css votedisk
[root@linuxidc ~]# ocrconfig -delete +OCR1
[root@linuxidc ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3116
Available space (kbytes) : 259004
ID : 1718040627
Device/File Name : +OCR2
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
Located 3 voting disk(s).
节点 2 的 ocrcheck 输出也没问题，略。2.1.6 重启整个集群
两个节点 root 执行：
crsctl stop crs && crsctl start crs
2.1.7 将旧的磁盘组 drop 掉
检查参数文件：
SQL> show parameter pfile
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
spfile string +OCR2/jyrac-scan/asmparameterf
ile/registry.253.920008067检查磁盘组状态：
Select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;最后将旧的磁盘组 mount 并 drop 掉：
alter diskgroup OCR1 mount;
drop diskgroup OCR1 including contents;
SQL> alter diskgroup OCR1 mount;
Diskgroup altered.
SQL> drop diskgroup OCR1 including contents;
Diskgroup dropped.注：旧的磁盘组只允许在一个节点 mount，如果发现多个节点 mount，需要在其他节点 dismount，否则会删除不掉旧的磁盘组。
2.2 迁移 DATA1
2.2.1 数据磁盘组 DATA1 添加
确定 /dev/asm-diskl，/dev/asm-diskm，/dev/asm-diskn 对应的是新加的 5G 大小的 3 个盘
用于数据磁盘组 DATA1 添加
alter diskgroup DATA1 add disk ‘/dev/asm-diskl’ rebalance power 7;
alter diskgroup DATA1 add disk ‘/dev/asm-diskm’, ‘/dev/asm-diskn’ rebalance power 11;
SQL> alter diskgroup DATA1 add disk '/dev/asm-diskl' rebalance power 7;
Diskgroup altered.
SQL> alter diskgroup DATA1 add disk '/dev/asm-diskm', '/dev/asm-diskn' rebalance power 11;
Diskgroup altered.这里说明可以单个添加，也可以多个添加。
2.2.2 查询同步状态
set linesize 200
col NAME for a10
select name,allocation_unit_size,state,type,free_mb,required_mirror_free_mb req_mi_fr_mb,usable_file_mb
from v$asm_diskgroup;2.2.3 删除原存储的 LUN
所有磁盘添加进 ASM 磁盘组后，删除原存储的 LUN。
查询 LUN path 与 asm_disk_name 的对应关系
select name,path from v$asm_disk;
SQL> select name,path from v$asm_disk
NAME PATH
---------- ------------------------------------------------------------
/dev/asm-diskc
/dev/asm-diskd
/dev/asm-disko
/dev/asm-diskb
OCR2_0001 /dev/asm-diskj
OCR2_0000 /dev/asm-diski
DATA1_0003 /dev/asm-diskl
DATA1_0004 /dev/asm-diskm
DATA1_0005 /dev/asm-diskn
OCR2_0002 /dev/asm-diskk
FRA1_0000 /dev/asm-diskh
DATA1_0001 /dev/asm-diskf
DATA1_0002 /dev/asm-diskg
DATA1_0000 /dev/asm-diske
14 rows selected.确定是 DATA1_0000，DATA1_0001，DATA1_0002
删除磁盘组 DATA1 中旧存储对应的 LUN：
alter diskgroup DATA1 drop disk DATA1_0000 rebalance power 7;
alter diskgroup DATA1 drop disk DATA1_0001,DATA1_0002 rebalance power 7;2.2.4 确认存储置换完成
select disk_number,path,total_mb,free_mb from v$asm_disk;
DISK_NUMBER PATH TOTAL_MB FREE_MB
----------- ------------------------------------------------------------ ---------- ----------
0 /dev/asm-diskc 0 0
1 /dev/asm-diskd 0 0
2 /dev/asm-disko 0 0
3 /dev/asm-diske 0 0
4 /dev/asm-diskf 0 0
5 /dev/asm-diskg 0 0
6 /dev/asm-diskb 0 0
1 /dev/asm-diskj 1024 715
0 /dev/asm-diski 1024 717
3 /dev/asm-diskl 5120 4502
4 /dev/asm-diskm 5120 4499
5 /dev/asm-diskn 5120 4501
2 /dev/asm-diskk 1024 714
0 /dev/asm-diskh 5120 4790
14 rows selected.2.3��移 FRA1
FRA1 磁盘组和 DATA1 磁盘组的添加一样，
alter diskgroup FRA1 add disk '/dev/asm-disko' rebalance power 7;
alter diskgroup FRA1 drop disk FRA1_0000 rebalance power 7;
select disk_number,path,total_mb,free_mb from v$asm_disk;最后确定所有盘都置换完成：
SQL> select disk_number,path,total_mb,free_mb from v$asm_disk;
DISK_NUMBER PATH TOTAL_MB FREE_MB
----------- ------------------------------------------------------------ ---------- ----------
0 /dev/asm-diskc 0 0
1 /dev/asm-diskd 0 0
3 /dev/asm-diske 0 0
4 /dev/asm-diskf 0 0
5 /dev/asm-diskg 0 0
6 /dev/asm-diskb 0 0
1 /dev/asm-diskj 1024 715
0 /dev/asm-diski 1024 717
1 /dev/asm-disko 5120 4943
3 /dev/asm-diskl 5120 4502
4 /dev/asm-diskm 5120 4499
5 /dev/asm-diskn 5120 4501
2 /dev/asm-diskk 1024 714
0 /dev/asm-diskh 5120 4965
14 rows selected.
SQL> r
1* select disk_number,path,total_mb,free_mb from v$asm_disk
DISK_NUMBER PATH TOTAL_MB FREE_MB
----------- ------------------------------------------------------------ ---------- ----------
0 /dev/asm-diskc 0 0
1 /dev/asm-diskd 0 0
2 /dev/asm-diskh 0 0
3 /dev/asm-diske 0 0
4 /dev/asm-diskf 0 0
5 /dev/asm-diskg 0 0
6 /dev/asm-diskb 0 0
1 /dev/asm-diskj 1024 715
0 /dev/asm-diski 1024 717
1 /dev/asm-disko 5120 4790
3 /dev/asm-diskl 5120 4502
4 /dev/asm-diskm 5120 4499
5 /dev/asm-diskn 5120 4501
2 /dev/asm-diskk 1024 714
14 rows selected.最终确定要替换的 LUN 都没有数据了，主机侧就可以操作了。
2.4 测试
查看集群各资源状态：
crsctl stat res -t查看 OCR 信息：
ocrcheck && crsctl query css votedisk查询 ASM 实例的 spfile 文件：
sqlplus / as sysasm <<EOF
show parameter pfile
EOF查询之前的业务数据：
-- 业务查询 SQL 1
select * from t1, t2 where t1.id = t2.t1_id and t1.n = 19;
-- 业务查询 SQL 2
select * from t1, t2 where t1.id = t2.t1_id;至此，整个 RAC 环境更换存储的实验就全部完成了。
更多 Oracle 相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=12
本文永久更新链接地址：http://www.linuxidc.com/Linux/2016-08/134555.htm
正文完
星哥玩云-微信公众号
