共计 12528 个字符，预计需要花费 32 分钟才能阅读完成。
实验环境：
CentOS 6.5 x64 RHCS GFS
配置说明: 本文出自: http://www.linuxidc.com
1. 通过 Openfiler 实现 iscsi 共享存储
2. 通过 VMware ESXi5 虚拟 fence 实现 fence 功能。
3. 结合 Centos 6.5 vmware-fence-soap 实现 RHCS fence 设备功能。
4. 通过搭建 RHCS 实验环境测试 GFS2 功能。
RHCS 高可用部署以及 GFS2/CLVM 集群文件系统应用 http://www.linuxidc.com/Linux/2014-07/104786.htm
红帽子 RHCS 套件安装与配置 http://www.linuxidc.com/Linux/2014-06/102770.htm
RHCS+GFS(红帽 HA+GFS) http://www.linuxidc.com/Linux/2014-04/99574.htm
基于 RHCS+iSCSI+CLVM 实现 Web 服务的共享存储集群架构 http://www.linuxidc.com/Linux/2013-05/84888.htm
RHCS 多节点部署应用企业环境 http://www.linuxidc.com/Linux/2013-05/84456.htm
RedHat 5.8 x64 RHCS Oracle 10gR2 HA 实践配置 http://www.linuxidc.com/Linux/2013-03/81589.htm
RHCS 基于 Java 的 HA 双机热备 http://www.linuxidc.com/Linux/2012-12/76587.htm
图解 RHCS 的安装配置与应用 http://www.linuxidc.com/Linux/2012-11/75111.htm
一、准备基础环境
1. 网络环境准备
node01,node02 节点
# cat /etc/hosts
192.168.0.181 node01.abc.com node01
192.168.0.182 node01.abc.com node02
2. 配置 YUM 安装源
(1) 挂载光盘 ISO
# mount /dev/cdrom /mnt
(2) 配置 YUM 客户端
说明: 通过本地光盘做为 yum 安装源。
# vi /etc/yum.repos.d/rhel.repo
[rhel]
name=rhel6
baseurl=file:///mnt
enabled=1
gpgcheck=0
(3) openfiler iscsi 存储配置
配置略, 规划磁盘空间如下:
qdisk 100MB
data 150GB
(4) node01,node02 挂载存储
# yum install iscsi-initiator-utils
# chkconfig iscsid on
# service iscsid start
# iscsiadm -m discovery -t sendtargets -p 192.168.0.187
192.168.0.187:3260,1 iqn.2006-01.com.openfiler:tsn.dea898a36535
# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.dea898a36535 -p 192.168.0.187 –l
二、RHCS 软件包的安装
1. 在 node01 上安装 luci 及 RHCS 软件包
1) node01（管理节点）安装 RHCS 软件包，luci 是管理端软件包，只在管理端安装。
yum -y install luci cman odcluster ricci gfs2-utils rgmanager lvm2-cluster
2) node02 安装 RHCS 软件包
yum -y install cman odcluster ricci gfs2-utils rgmanager lvm2-cluster
3) node01, node02 更改个节点 ricci 用户密码
passwd ricci
4) 配置 RHCS 服务开机启动
chkconfig ricci on
chkconfig rgmanager on
chkconfig cman on
service ricci start
service rgmanager start
service cman start
# 启动过程如下:
正在启动 oddjobd：[确定]
generating SSL certificates… done
Generating NSS database… done
启动 ricci：[确定]
Starting Cluster Service Manager: [确定]
Starting cluster:
Checking if cluster has been disabled at boot… [确定]
Checking Network Manager… [确定]
Global setup… [确定]
Loading kernel modules… [确定]
Mounting configfs… [确定]
Starting cman… xmlconfig cannot find /etc/cluster/cluster.conf
[失败]
Stopping cluster:
Leaving fence domain… [确定]
Stopping gfs_controld… [确定]
Stopping dlm_controld… [确定]
Stopping fenced… [确定]
Stopping cman… [确定]
Unloading kernel modules… [确定]
Unmounting configfs… [确定]
#
2. 在 node01 管理节点上安装启动 luci 服务
1) 启动 luci 服务
chkconfig luci on
service luci start
Adding following auto-detected host IDs (IP addresses/domain names), corresponding to `node01′ address, to the configuration of self-managed certificate `/var/lib/luci/etc/cacert.config’ (you can change them by editing `/var/lib/luci/etc/cacert.config’, removing the generated certificate `/var/lib/luci/certs/host.pem’ and restarting luci):
(none suitable found, you can still do it manually as mentioned above)
Generating a 2048 bit RSA private key
writing new private key to ‘/var/lib/luci/certs/host.pem’
Start luci… [确定]
Point your web browser to https://node01:8084 (or equivalent) to access luci
2) 配置管理地址, RHCS6 版本采用 root 用户密码登录。
https://node01:8084
root/111111
三、RHCS 集群配置
1. 添加集群
登录进管理界面，点击 Manage Clusters –> Create 填入如下内容：
Cluster Name: gfs
NodeName Password RicciHostname Ricci Port
node01.abc.com 111111 node01.abc.com 11111
node02.abc.com 111111 node01.abc.com 11111
选中如下选项，然后提交
Use locally installed packages.
说明: 这步会生成集群配置文件 /etc/cluster/cluster.conf
2. 添加 Fence Devices
说明:
RHCS 要实现完整的集群功能，必须要实现 fence 功能。由于非物理服务器配置等条件限制，特使用 VMware ESXi5.X 的虚拟 fence 来实现 fence 设备的功能。
正是由于有了 fence 设备可以使用，才得以完整测试 RHCS 功能。
(1) 登录进管理界面，点击 cluster-> Fence Devices->
(2) 选择 ”Add 选择 VMware Fencing（SOAP Interface）
(3)Name “ESXi_fence”
(4)IP Address or Hostname “192.168.0.21”（ESXi 地址）
(5)Login “root”
(6)Password “111111”
3. 节点绑定 Fence 设备
添加节点一 fence
1) 点击 node01.abc.com 节点，Add Fence Method，这里填 node01_fence;
2) 添加一个 fence instance, 选择 ”ESXi_fence” VMware Fencing（SOAP Interface）
3) VM NAME “kvm_node01”
4) VM UUID “564d6fbf-05fb-1dd1-fb66-7ea3c85dcfdf” 选中 ssl
说明: VMNAME: 虚拟机名称，VM UUID: 虚拟机.vmx 文件中 ”
uuid.location” 值, 采用下面的字符串的格式。
# /usr/sbin/fence_vmware_soap -a 192.168.0.21 -z -l root -p xjj876543 -n kvm_node2 -o list
kvm_node2,564d4c42-e7fd-db62-3878-57f77df2475e
kvm_node1,564d6fbf-05fb-1dd1-fb66-7ea3c85dcfdf
添加节点二 fence
1) 点击 node02.abc.com 节点，Add Fence Method，这里填 node02_fence;
2) 添加一个 fence instance, 选择 ”ESXi_fence” VMware Fencing（SOAP Interface）
3) VM NAME “kvm_node02”
4) VM UUID “564d4c42-e7fd-db62-3878-57f77df2475e” 选中 ssl
# 手动测试 fence 功能示例:
# /usr/sbin/fence_vmware_soap -a 192.168.0.21 -z -l root -p xjj876543 -n kvm_node02 -o reboot
Status: ON
选项：
-o : list,status,reboot 等参数
4. 添加 Failover Domains 配置
Name “gfs_failover”
Prioritized
Restricted
node01.abc.com 1
node02.abc.com 1
5. 配置 GFS 服务
(1) GFS 服务配置
分别在 node01,node02 启动 CLVM 的集成 cluster 锁服务
lvmconf –enable-cluster
chkconfig clvmd on
service clvmd start
Activating VG(s): No volume groups found [OK]
(2) 在任意一节点对磁盘进行分区, 划分出 sdc1。然后格式化成 gfs2.
node01 节点上：
# pvcreate /dev/sdc1
Physical volume “/dev/sdc1” successfully created
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg_node01 lvm2 a– 39.51g 0
/dev/sdc1 lvm2 a– 156.25g 156.25g
# vgcreate gfsvg /dev/sdc1
Clustered volume group “gfsvg” successfully created
# lvcreate -l +100%FREE -n data gfsvg
Logical volume “data” created
node02 节点上：
# /etc/init.d/clvmd start
(3) 格式化 GFS 文件系统
node01 节点上：
[root@node01 ~]# mkfs.gfs2 -p lock_dlm -t gfs:gfs2 -j 2 /dev/gfsvg/data
This will destroy any data on /dev/gfsvg/data.
It appears to contain: symbolic link to `../dm-2′
Are you sure you want to proceed? [y/n] y
Device: /dev/gfsvg/data
Blocksize: 4096
Device Size 156.25 GB (40958976 blocks)
Filesystem Size: 156.25 GB (40958975 blocks)
Journals: 2
Resource Groups: 625
Locking Protocol: “lock_dlm”
Lock Table: “gfs:gfs2”
UUID: e28655c6-29e6-b813-138f-0b22d3b15321
说明:
gfs:gfs2 这个 gfs 就是集群的名字，gfs2 是定义的名字，相当于标签。
- j 是指定挂载这个文件系统的主机个数，不指定默认为 1 即为管理节点的。
这里实验有两个节点
6. 挂载 GFS 文件系统
node01,node02 上创建 GFS 挂载点
# mkdir /vmdata
(1)node01,node02 手动挂载测试，挂载成功后，创建文件测试集群文件系统情况。
# mount.gfs2 /dev/gfsvg/data /vmdata
(2) 配置开机自动挂载
# vi /etc/fstab
/dev/gfsvg/data /vmdata gfs2 defaults 0 0
[root@node01 vmdata]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_node01-lv_root 36G 3.8G 30G 12% /
tmpfs 1.9G 32M 1.9G 2% /dev/shm
/dev/sda1 485M 39M 421M 9% /boot
/dev/gfsvg/data 157G 259M 156G 1% /vmdata
7. 配置表决磁盘
说明:
# 表决磁盘是共享磁盘，无需要太大，本例采用 /dev/sdc1 100MB 来进行创建。
[root@node01 ~]# fdisk -l
Disk /dev/sdb: 134 MB, 134217728 bytes
5 heads, 52 sectors/track, 1008 cylinders
Units = cylinders of 260 * 512 = 133120 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x80cdfae9
Device Boot Start End Blocks Id System
/dev/sdb1 1 1008 131014 83 Linux
(1) 创建表决磁盘
[root@node01 ~]# mkqdisk -c /dev/sdb1 -l myqdisk
mkqdisk v0.6.0
Writing new quorum disk label ‘myqdisk’ to /dev/sdc1.
WARNING: About to destroy all data on /dev/sdc1; proceed [N/y] ? y
Initializing status block for node 1…
Initializing status block for node 2…
Initializing status block for node 3…
Initializing status block for node 4…
Initializing status block for node 5…
Initializing status block for node 6…
Initializing status block for node 7…
Initializing status block for node 8…
Initializing status block for node 9…
Initializing status block for node 10…
Initializing status block for node 11…
Initializing status block for node 12…
Initializing status block for node 13…
Initializing status block for node 14…
Initializing status block for node 15…
Initializing status block for node 16…
(2) 查看表决磁盘信息
[root@node01 ~]# mkqdisk -L
mkqdisk v3.0.12.1
/dev/block/8:17:
/dev/disk/by-id/scsi-14f504e46494c455242553273306c2d4b72697a2d544e6b4f-part1:
/dev/disk/by-path/ip-192.168.0.187:3260-iscsi-iqn.2006-01.com.openfiler:tsn.dea898a36535-lun-0-part1:
/dev/sdb1:
Magic: eb7a62c2
Label: myqdisk
Created: Thu Jan 1 23:42:00 2015
Host: node02.abc.com
Kernel Sector Size: 512
Recorded Sector Size: 512
(3) 配置表决磁盘 qdisk
# 进入管理界面 Manage Clusters –> gfs –> Configure –> QDisk
Device : /dev/sdc1
Path to program : ping -c3 -t2 192.168.0.253
Interval : 3
Score : 2
TKO : 10
Minimum Score : 1
# 点击 apply
(4) 启动 qdisk 服务
chkconfig qdiskd on
service qdiskd start
clustat -l
[root@node01 ~]# clustat -l
Cluster Status for gfs @ Thu Jan 1 23:50:53 2015
Member Status: Quorate
Member Name ID Status
—— —- —- ——
node01.abc.com 1 Online, Local
node02.abc.com 2 Online
/dev/sdb1 0 Online, Quorum Disk
[root@node01 ~]#
8. 测试 GFS
1）node02 节点上执行
# echo c > /proc/sysrq-trigger
2）node01 节点上查看日志记录
# tail -f /var/log/messages
Jan 2 01:37:47 node01 ricci: startup succeeded
Jan 2 01:37:47 node01 rgmanager[2196]: I am node #1
Jan 2 01:37:47 node01 rgmanager[2196]: Resource Group Manager Starting
Jan 2 01:37:47 node01 rgmanager[2196]: Loading Service Data
Jan 2 01:37:49 node01 rgmanager[2196]: Initializing Services
Jan 2 01:37:49 node01 rgmanager[2196]: Services Initialized
Jan 2 01:37:49 node01 rgmanager[2196]: State change: Local UP
Jan 2 01:37:49 node01 rgmanager[2196]: State change: node02.abc.com UP
Jan 2 01:37:52 node01 polkitd[3125]: started daemon version 0.96 using authority implementation `local’ version `0.96′
Jan 2 01:37:52 node01 rtkit-daemon[3131]: Sucessfully made thread 3129 of process 3129 (/usr/bin/pulseaudio) owned by ’42’ high priority at nice level -11.
Jan 2 01:40:52 node01 qdiskd[1430]: Assuming master role
Jan 2 01:40:53 node01 qdiskd[1430]: Writing eviction notice for node 2
Jan 2 01:40:54 node01 qdiskd[1430]: Node 2 evicted
Jan 2 01:40:55 node01 corosync[1378]: [TOTEM] A processor failed, forming new configuration.
Jan 2 01:40:57 node01 corosync[1378]: [QUORUM] Members[1]: 1
Jan 2 01:40:57 node01 corosync[1378]: [TOTEM] A processor joined or left the membership and a new membership was formed.
Jan 2 01:40:57 node01 kernel: dlm: closing connection to node 2
Jan 2 01:40:57 node01 corosync[1378]: [CPG] chosen downlist: sender r(0) ip(192.168.0.181) ; members(old:2 left:1)
Jan 2 01:40:57 node01 corosync[1378]: [MAIN] Completed service synchronization, ready to provide service.
Jan 2 01:40:57 node01 kernel: GFS2: fsid=gfs:gfs2.1: jid=0: Trying to acquire journal lock…
Jan 2 01:40:57 node01 fenced[1522]: fencing node node02.abc.com
Jan 2 01:40:57 node01 rgmanager[2196]: State change: node02.abc.com DOWN
Jan 2 01:41:11 node01 fenced[1522]: fence node02.abc.com success
Jan 2 01:41:12 node01 kernel: GFS2: fsid=gfs:gfs2.1: jid=0: Looking at journal…
Jan 2 01:41:12 node01 kernel: GFS2: fsid=gfs:gfs2.1: jid=0: Done
Jan 2 01:41:30 node01 corosync[1378]: [TOTEM] A processor joined or left the membership and a new membership was formed.
Jan 2 01:41:30 node01 corosync[1378]: [QUORUM] Members[2]: 1 2
Jan 2 01:41:30 node01 corosync[1378]: [QUORUM] Members[2]: 1 2
Jan 2 01:41:30 node01 corosync[1378]: [CPG] chosen downlist: sender r(0) ip(192.168.0.181) ; members(old:1 left:0)
Jan 2 01:41:30 node01 corosync[1378]: [MAIN] Completed service synchronization, ready to provide service.
Jan 2 01:41:38 node01 qdiskd[1430]: Node 2 shutdown
Jan 2 01:41:50 node01 kernel: dlm: got connection from 2
Jan 2 01:41:59 node01 rgmanager[2196]: State change: node02.abc.com UP
说明：fence 功能正常，期间 GFS 文件系统正常。
9. 配置文件
cat /etc/cluster/cluster.conf
<?xml version=”1.0″?>
<cluster config_version=”9″ name=”gfs”>
<clusternodes>
<clusternode name=”node01.abc.com” nodeid=”1″>
<fence>
<method name=”node01_fence”>
<device name=”ESXi_fence” port=”kvm_node1″ ssl=”on” uuid=”564d6fbf-05fb-1dd1-fb66-7ea3c85dcfdf”/>
</method>
</fence>
</clusternode>
<clusternode name=”node02.abc.com” nodeid=”2″>
<fence>
<method name=”node02_fence”>
<device name=”ESXi_fence” port=”kvm_node2″ ssl=”on” uuid=”564d4c42-e7fd-db62-3878-57f77df2475e”/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes=”3″/>
<fencedevices>
<fencedevice agent=”fence_vmware_soap” ipaddr=”192.168.0.21″ login=”root” name=”ESXi_fence” passwd=”111111″/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name=”gfs_failover” nofailback=”1″ ordered=”1″>
<failoverdomainnode name=”node01.abc.com” priority=”1″/>
<failoverdomainnode name=”node02.abc.com” priority=”1″/>
</failoverdomain>
</failoverdomains>
</rm>
<quorumd device=”/dev/sdb1″ min_score=”1″>
<heuristic interval=”3″ program=”ping -c3 -t2 192.168.0.253″ score=”2″ tko=”10″/>
</quorumd>
</cluster>
更多 CentOS 相关信息见 CentOS 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=14
