Prerequisites:
- 2 OSS nodes connected via SAS over multiple paths to their own JBOD shared storage (minimum eight drives)
- 2 MDS nodes connected via SAS over multiple paths to their own JBOD shared storage (minimum four drives)
- Git is installed on the head node
Requirements:
- The Bright version is 8.0.
- The MDS nodes must be setup for active-passive high availability.
- The backend storage for the MDS nodes will be software RAID 10.
- The OSS nodes must be setup for active-active high availability.
- The backend storage for the OSS nodes will be ZFS raidz2 (RAID 6) pools (at least one pool for each node).
Create MDS software image
1. Clone the default-image to initialize the MDS software image.
cmsh % softwareimage % clone default-image lustre-mds-image % commit
2. Clone the lustre-release git repository onto the head node:
cd /root git clone git://git.hpdd.intel.com/fs/lustre-release.git
3. Use the cloned repository to create the RPMs required for the MDS nodes:
cd /root/lustre-release sh ./autogen.sh ./configure make rpms
4. Install the new RPMs onto the MDS software image:
yum --installroot=/cm/images/lustre-mds-image localinstall *.rpm
5. Set the software image's kernelversion to the newly installed lustre kernel:
cmsh -c "softwareimage use lustre-mds-image; set kernelversion <new-lustre-kernel>; commit"
6. Install device mapper multipath onto image:
yum --installroot=/cm/images/lustre-mds-image install device-mapper-multipath
Create category for MDS nodes
cmsh % category % clone default lustre-mds % set softwareimage lustre-mds-image % set installbootrecord yes% clear roles % commit % device -n mds01..mds02 (set category lustre-mds; commit)
Now provision and boot the two MDS nodes.
Setup MDS backend storage
1. Log into mds01.
2. Setup multipath:
mpathconf --enable --with_multipathd y
3. Copy /etc/multipath.conf and /etc/multipath/bindings to mds02 so that the mappings are consistent:
scp /etc/multipath.conf mds02:/etc/multipath.conf scp /etc/multipath/bindings mds02:/etc/multipath/bindings
4. Create software RAID 10 volume using multipath mappings and save its configuration to file:
mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/mapper/mpath[a-d] mdadm --detail --scan --verbose >> /etc/mdadm.conf scp /etc/mdadm.conf mds02:/etc/mdadm.conf
5. Create the mgs and mdt using /dev/md0:
mkfs.lustre --fsname lustre00 --mdt --mgs --failnode=<NID-of-mds02> /dev/md0
If Lustre is to communicate solely over Infiniband and if there are multiple types of network interfaces on the Lustre server and client nodes, then add the following modprobe configuration to each of those nodes; otherwise, skip to step 6.
# cat /etc/modprobe.d/lustre.conf options lnet networks=o2ib
Because Lustre may actually choose to use Ethernet for its NIDs, the above configuration is necessary to force Lustre to only use Infiniband. After adding the configuration, the Lustre nodes will need to be rebooted.
6. Mount the mdt:
mkdir /mnt/mdt; mount -t lustre /dev/md0 /mnt/mdt
Now go back to the head node.
Create failovergroup for MDS nodes
1. Update the software image for the MDS nodes:
cmsh -c "device use mds01; grabimage -w"
2. Create the Bright failover group:
cmsh % partition use base % failovergroups % add mdsgroup % set nodes mds01..mds02 % set automaticfailoveraftergracefulshutdown yes% set mountscript <path-to-script-with-contents-shown-below>% set unmountscript <path-to-script-with-contents-shown-below>% commit
Here are the contents of the mount script, which will be used when a passive MDS becomes active:
#!/bin/sh/bin/systemctl start multipathd.service || { echo "Multipath not started." ; exit 1; }/sbin/mdadm --assemble --scan || { echo "Software RAID not started." ; exit 1; }/bin/mount /mnt/mdt || { echo "Cannot mount mdt." ; exit 1; }
Here are the contents of the unmount script, which will be used when an active MDS becomes passive:
#!/bin/sh/bin/umount -f /mnt/mdt || { echo "Unmount failed." ; exit 1; }/sbin/mdadm --stop --scan || { echo "Software RAID not stopped." ; exit 1; }/bin/systemctl stop multipathd.service || { echo "Multipath not stopped." ; exit 1; }/sbin/multipath -F || { echo "Paths not flushed." ; exit 1; }
3. Add the multipathd service to the MDS category. It will only be run on the active MDS:
% category use lustre-mds % services % add multipathd % set runif active % set autostart yes% set monitoring yes% commit
Create software images for OSS nodes
1. Clone the default-image to initialize the software image for oss01:
cmsh % softwareimage % clone default-image lustre-oss01-image % commit
2. Use the lustre-release git repository to create the RPMs for the OSS nodes:
cd /root/lustre-release make clean rm -f *.rpm sh ./autogen.sh ./configure --disable-ldiskfsmake rpms
3. Install the new RPMs onto the oss01 software image:
yum --installroot=/cm/images/lustre-oss01-image localinstall *.rpm
4. Set the software image's kernelversion to the newly installed lustre kernel:
cmsh -c "softwareimage use lustre-oss01-image; set kernelversion <new-lustre-kernel>; commit"
5. Install device mapper multipath onto image:
yum --installroot=/cm/images/lustre-oss01-image install device-mapper-multipath
6. Install zfs onto image and enable the appropriate services for boot:
yum --installroot=/cm/images/lustre-oss01-image install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
Enable the [zfs-kmod] repository in /cm/images/lustre-oss01-image/etc/yum.repos.d/zfs.repo.
yum --installroot=/cm/images/lustre-oss01-image install zfs libzfs2-devel kmod-spl-devel kmod-zfs-devel chroot /cm/images/lustre-oss01-image systemctl enable zfs-import-scan zfs-mount zfs-share zfs-zed zfs.target systemctl disable zfs-import-cache
Modify /lib/systemd/system/zfs-import-scan.service to have the following contents:
[Unit]Description=Import ZFS pools by device scanning DefaultDependencies=no Requires=systemd-udev-settle.service After=systemd-udev-settle.service After=cryptsetup.targetConditionPathExists=!/etc/zfs/zpool.cache [Service]Type=oneshot RemainAfterExit=yesExecStartPre=/sbin/modprobe zfs ExecStart=/usr/local/bin/zpool-import.shExecStop=/usr/local/bin/zpool-export.sh[Install]WantedBy=zfs-mount.service WantedBy=zfs.target
The /usr/local/bin/zpool-import.sh script will import a specified set of zpools on each OSS node, and that set of zpools will be different on each. This is the normal HA state, which we will call State 0. Create this script to have the following contents in this example:
#!/bin/bash/sbin/zpool import -o cachefile=none ostpool01
Create the complementing /usr/local/bin/zpool-export.sh as follows:
#!/bin/bash/sbin/zpool export -f ostpool01
7. Clone the oss01 image to create the oss02 image:
cmsh -c "softwareimage clone lustre-oss01-image lustre-oss02-image; commit"
Now change /usr/local/bin/zpool-import.sh on lustre-oss02-image as follows:
#!/bin/bash/sbin/zpool import -o cachefile=none ostpool02
Create the complementing /usr/local/bin/zpool-export.sh as follows:
#!/bin/bash/sbin/zpool export -f ostpool02
Create categories for OSS nodes
cmsh % category % clone lustre-mds lustre-oss01 % set softwareimage lustre-oss01-image % services % add multipathd % set autostart yes% set monitoring yes% .. % clone lustre-oss01 lustre-oss02 % set softwareimage lustre-oss02-image % commit % device use oss01; set category lustre-oss01; commit % device use oss02; set category lustre-oss02; commit
Now provision and boot the two OSS nodes.
Setup OSS backend storage
1. Log into oss01.
2. Setup multipath:
mpathconf --enable --with_multipathd y
3. Restart multipathd:
systemctl restart multipathd
4. Copy /etc/multipath/bindings to oss02 for consistency:
scp /etc/multipath/bindings oss02:/etc/multipath/bindings
5. Back on oss01, create your raidz2 zpools. For this example, the shared storage only has 8 drives so a raidz2 pool of 4 drives will be created:
zpool create -fo cachefile=none ostpool01 raidz2 /dev/mapper/mpath[a-d]
6. Create the osts using the created zpools:
mkfs.lustre --ost --backfstype=zfs --fsname=lustre00 --index=0 --mgsnode=<NID-of-mds01> --mgsnode=<NID-of-mds02> --servicenode=<NID-of-oss01> --servicenode=<NID-of-oss02> ostpool01/ost01 zfs set recordsize=1M ostpool01/ost01
7. Mount the osts:
mkdir /mnt/ost01; mount -t lustre ostpool01/ost01 /mnt/ost01
8. On oss02, create the remaining raidz2 pools. For this example, there are only 4 unallocated drives left so there is only one pool left to create:
zpool create -fo cachefile=none ostpool02 raidz2 /dev/mapper/mpath[e-h]
9. Create the osts using the created zpools:
mkfs.lustre --ost --backfstype=zfs --fsname=lustre00 --index=0 --mgsnode=<NID-of-mds01> --mgsnode=<NID-of-mds02> --servicenode=<NID-of-oss01> --servicenode=<NID-of-oss02> ostpool02/ost02 zfs set recordsize=1M ostpool02/ost02
10. Mount the osts:
mkdir /mnt/ost02; mount -t lustre ostpool02/ost02 /mnt/ost02
11. On the head node, create mounts for the osts:
cmsh % category use lustre-oss01 % fsmounts % add /mnt/ost01 % set device ostpool01/ost01 % set filesystem lustre % set mountoptions "rw,_netdev"% category use lustre-oss02 % add /mnt/ost02 % set device ostpool02/ost02 % set filesystem lustre % set mountoptions "rw,_netdev"% commit
12. Make sure both images are up-to-date:
% device use oss01; grabimage -w% device use oss02; grabimage -w
Create failovergroup for OSS nodes
Create the Bright failover group:
cmsh % partition use base % failovergroups % add ossgroup % set nodes oss01..oss02 % set automaticfailoveraftergracefulshutdown yes% set prefailoverscript <path-to-script-with-contents-shown-below>% set postfailoverscript <path-to-script-with-contents-shown-below>% commit
The contents of the prefailover script should be:
#!/bin/sh# Do not import zpools if missing multipath mpaths[ $(/usr/sbin/multipath -ll | grep -c mpath) -ne 8 ] && { echo "FATAL: Not seeing all multipath mpaths." ; exit 1; }# Import appropriate zpools on each OSS node/usr/local/bin/zpool-import.sh || { echo "FATAL: Cannot import usual zpools." ; exit 1; }
NOTE: The 8 multipath paths should be altered in this script to be the actual number of multipaths on your system.
The contents of the postfailover script should be:
#!/usr/bin/pythonimport osimport subprocess# Read current state f = open('/var/spool/cmd/state') state = f.readline() f.close()# Get the hostname command = subprocess.Popen('hostname', stdout=subprocess.PIPE);(hostname, err) = command.communicate() hostname = hostname.rstrip("\n")# function to check if the counter part is onlinedef isonline (nodename): # here we do the ping check to determine whether the other node is up or down response = os.system("ping -c 1 " + nodename);if response != 0: print "passive node is down";# passive node down# check if the multipath is correct command = subprocess.Popen('/usr/sbin/multipath -ll | grep -c mpath' , stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True);(multipathcount, err) = command.communicate() multipathcount = multipathcount.rstrip("\n")print multipathcount if int(multipathcount) != 8: print "FATAL: Not seeing all multipath mpaths." exit(1)#passive node down and mutilpaths are present, run the zpool-import-global.sh response = os.system("/usr/local/bin/zpool-import-global.sh");if state == 'SLAVEACTIVE': print "SLAVEACTIVE";if hostname=='oss01': print "oss01" isonline("oss02");else: # I'm oss02print "oss02" isonline("oss01")if state == 'SLAVEPASSIVE': print "SLAVEPASSIVE";
NOTE: The 8 multipath paths should be altered in this script to be the actual number of multipaths on your system.
Note that the postfailover script calls /usr/local/bin/zpool-import-global.sh, which will import and mount the failed OSS node's zpools and OSTs, respectively. For this example, the contents of this script on oss01 should be:
#!/bin/bash/sbin/zpool import -f -o cachefile=none ostpool02 /usr/bin/mount -t lustre ostpool02/ost02 /mnt/ost02
And the contents of this script on oss02 should be:
#!/bin/bash/sbin/zpool import -f -o cachefile=none ostpool01 /usr/bin/mount -t lustre ostpool01/ost01 /mnt/ost01
If the check has any timeout issues, then add the following advanced config setting to cmd.conf:
AdvancedConfig = { "FailoverPowerRetries=120" }
The actual number of retries can be lowered, as desired.
Create healthcheck for "passive" OSS node
The above failovergroup defines an active-passive HA scenario for the OSS nodes. To make the scenario active-active, a healthcheck can be added to handle when the passive OSS node of that failovergroup crashes.
1. Create the script, /cm/local/apps/cmd/scripts/healthchecks/check_ha_passive.sh, with these contents:
#!/bin/bash . /etc/profile.d/modules.sh module load cmsh # Failover has already occurred; OSTs are already imported and mounted; ExitFAILEDBEFORE="/var/spool/cmd/hafailed"if [ -e "$FAILEDBEFORE" ]; thenecho FAIL exit 0fi# Determine passive nodeactiveOSS= passiveOSS= state= for oss in oss01 oss02; dostate=$(ssh -o StrictHostKeyChecking=no -q $oss cat /var/spool/cmd/state)if [ "$state" == "SLAVEACTIVE" ]; thenactiveOSS=$osselsepassiveOSS=$ossfidone# If both nodes are down, FAILif [ -z "$activeOSS" ] && [ -z "$passiveOSS" ]; thenecho FAIL exit 0fi# If active node is down, this scenario is handled by Bright failovergroup so we PASS hereif [ -z "$activeOSS" ]; thenecho PASS exit 0fi# here we do the ping check to determine whether the other node is up or downping -q -c1 $passiveOSS > /dev/null if [ $? -eq 0 ]; thenecho PASS else# Ensure that passive node is powered off before migrating zpools cmsh -c "device use $passiveOSS; power off" 2>&1 > /dev/null sleep 15# check if the multipath is correct on active nodeif [ $(ssh -o StrictHostKeyChecking=no -q $activeOSS /usr/sbin/multipath -ll | grep -c mpath) -ne 8 ]; thenecho FAIL exit 0fi# Now import zpoolsssh -o StrictHostKeyChecking=no -q $activeOSS /usr/local/bin/zpool-import-global.sh 2>&1 > /dev/null touch $FAILEDBEFOREecho FAIL # FAIL because admin show see that the passive node is downfi
NOTE: The 8 multipath paths should be altered in this script to be the actual number of multipaths on your system.
2. Create the healthcheck and assign it to the headnode(s):
cmsh % monitoring setup % show checkhapassive Parameter Value -------------------------------- ------------------------------------------------------------ Arguments Automatic reinitialize yes Class Failover Consolidator Description Disabled no Execution multiplexer <0 in submode> Fuzzy offset 0 Gap 0 Interval 5m Maximal age 0s Maximal samples 4096 Measurables 1 / 3194 Name checkhapassive Node execution filters <1 in submode> Notes <0 bytes> Offset 0s Only when idle no Revision Run in bash no Script /cm/local/apps/cmd/scripts/healthchecks/check_ha_passive.sh Timeout 60 Type HealthCheckScript When Timed % use checkhapassive % nodeexecutionfilters % show active\ head\ node Parameter Value -------------------------------- ------------------------------------------------ Filter Active Name Active head node Resources Active Revision Type MonitoringResourceExecutionFilter
Create the Lustre client RPMs
cd /root/lustre-release make clean rm -f *.rpm sh ./autogen.sh ./configure --disable-server --enable-clientmake rpms
The RPMs can now be installed onto whatever image you choose for nodes that will be mounting the Lustre filesystem. For instructions on creating that image and a Lustre client category and setting up nodes to be Lustre clients, please refer to that section of the Bright installation manual. The one exception will be that the default-image, not any Lustre server image created earlier in these procedures, should be used in cloning to create the Lustre client image; otherwise, unnecessary files and packages associated with multipath, software RAID, and zfs will end up on the client nodes.