Skip to main content
Ask Question
Installing BCM
Asked a question last year

How do I upgrade my cluster if I have a storage node that is being shared?

Where am I?

In Bright Computing, Inc. you can ask and answer questions and share your experience with others!

How do I update a cluster with a storage node that is shared?

Upgrading with a Storage Node

Special considerations must be made when upgrading Bright versions with a storage node internal to the cluster:

The problem with this kind of setup is that all of the nodes except for the head node(s) must be shut down before starting the upgrade.

But filesystems from the storage node are being shared, and they need to be available to the head node for the upgrade.

To solve this, we will temporarily get rid of the dependency of the storage nodes internal to the cluster.

Let's do it by example.

For this example, we use node003 (10.141.0.3) as a storage node. This storage node, when it is running, exports /cm/shared to the cluster from the local path of node003 at /storage/cm/shared:

Steps:

1. Shut down all of the nodes, except for the head node(s), and the storage node.

In this case, assuming we have nodes node001 and node002 as the other nodes on the cluster, we will shut down those nodes:

% cmsh
[head]% device
[head->device]% shutdown node001
[head->device]% shutdown node002

2. On the active head node, copy /cm/shared over to /cm/shared.local:

# mkdir –p /cm/shared.local
# cp –a /cm/shared/* /cm/shared.local

3. Remove the mount from Bright Cluster Manager, and then unmount the NFS filesystem /cm/shared:

Remove mount from Bright:

# cmsh
[head]% device use master
[head->device[head]]% fsmounts
[head->device[head]->fsmounts]% remove /cm/shared
[head->device[head]->fsmounts*]% commit
[head->device[head]->fsmounts]% quit
# umount /cm/shared

4. Bind-mount /cm/shared.local on /cm/shared

# mount --bind /cm/shared.local /cm/shared

To make it persistent, add the following line in /etc/fstab, after the autogenerated section:

# END AUTOGENERATED SECTION   -- DO NOT REMOVE
/cm/shared.local   /cm/shared   none  bind  0 0

set storage node install mode to “” if it is currently set to NOSYNC

# cmsh
[head]% device use node003
[head->device[node003]% set installmode ""
[head->device[node003*]% commit

5. Shut down your storage node

# cmsh
[head]% device
[head->device]% shutdown node003

6. Perform the upgrade according to the instructions in this article:

http://kb.brightcomputing.com/faq/index.php?action=artikel&cat=2&id=44924

7. Boot your storage node.

8. Mount the storage node’s shared filesystem, /cm/shared,  onto /cm/shared.storage37 on the headnode

Check  /etc/exports on node003 to make sure the filesystem is exported with rw permissions for the head node that will access it.

On the head node create /cm/shared.storage37:

# mkdir –p /cm/shared.storage37
The filesystem is added to fstab, after the autogenerated section:

10.141.0.3:/storage/cm/shared    /cm/shared.storage37    nfs    defaults   0 0

Mount the filesystem according to the fstab settings:

# mount -a

9. Synchronize /cm/shared.storage37 and /cm/shared.local

# rsync -avh /cm/shared.local/ /cm/shared.storage/28

10. Now unmount the /cm/shared bind mount:

# umount /cm/shared

Remove it from /etc/fstab on the head node. i.e., remove the entry for /cm/shared.local:

/cm/shared.local   /cm/shared   none  bind  0 0

Remove this entry in /etc/fstab on the head node too:

10.141.0.3:/storage/cm/shared    /cm/shared.storage37   nfs   defaults     0 0

11. Mount /cm/shared from the storage node to all the other nodes:

Using cmsh on the head node, mounting /cm/shared for the head node:

# cmsh
[head]% device use master
[head->device[head]% fsmounts
[head->device[head]->fsmounts% add /cm/shared
[head->device*[head*]->fsmounts*[/cm/shared*]% set device 10.141.0.3:/storage/cm/shared
[head->device*[head*]->fsmounts*[/cm/shared*]% set filesystem nfs
[head->device*[head*]->fsmounts*[/cm/shared*]% commit

Using cmsh on the head node, mounting /cm/shared for the regular nodes:

# cmsh
[head->category[default]% category use default
[head->category[default]% fsmounts
[head->category[default]->fsmounts]% add /cm/shared
[head->category*[default*]->fsmounts*[/cm/shared*]% set device 10.141.0.3:/storage/cm/shared
[head->category*[default*]->fsmounts*[/cm/shared*]% set filesystem nfs
[head->category*[default*]->fsmounts*[/cm/shared*]% commit

12. Boot up the rest of the nodes inside of your cluster