How can I flash update all my nodes (from Linux)?
(For update via DOS, see http://kb.brightcomputing.com/faq/index.php?action=artikel&cat=20&id=94 )
Some manufacturers like Dell provide a flash BIOS upgrade utility that is run from within Linux. Such a utility typically requires the node to reboot from the hard drive after running the utility, and only then will the upgrade be complete. It typically therefore does not work with the nodes of a cluster, because nodes by default do a PXE boot.
Because the flash upgrade utility is usually a binary, it is unclear how it works. The procedure to make it work described next is based on some commonsense guesswork. The manufacturer should be contacted to confirm how their utility works, before trying out the procedure described next.
The procedure described next should be done with care because a node that has a damaged BIOS may not function at all. Such a node is called a node that is "bricked" because it may be as much use as a brick for its intended purpose.
The procedure is based on the likelihood that the utility modifies something on the local drive, probably a service which is loaded on system startup.
The trouble with the firmware update utility doing that (a modifcation that is to run on startup) is that a regular node in the cluster normally PXE boots from a software image, and not from the regular node hard drive that the utility has modified. So this is why, in a default cluster, updating a flash bios from Linux will not succeed for regular nodes.
For such a case, the utility can however usually be made to work by simply setting the node to non-sync on the next install. For example:
cmsh -c "device use node001; set nextinstallmode nosync; commit"
After running the firmware installation utility, and if the node has the updated bios on it after reboot and if the node is behaving ok, following procedure can then install the firmware to the remaining nodes:
1. Make sure the firmware binary is on all nodes by placing it in your software image, or by using pcopy in cmsh.
Download the firmware binary and save it to /opt on the head node
[bright60 ~]# chmod 755 /C6220_BIOS_15R41_LN32_1.1.9.BIN
[bright60 ~]# cmsh
[bright60]% device pcopy -n node002..node200 /opt/C6220_BIOS_15R41_LN32_1.1.9.BIN /opt
[bright60]% device commit
2. set the nextinstallmode to NOSYNC. example:
[bright60]% device foreach -n node002..node200 (set nextinstallmode nosync)
[bright60]% device commit
3. Use pexec to call the firmware upgrade utility on all nodes. Example:
[bright60]% device pexec -n node002..node200 " /opt/C6220_BIOS_15R41_LN32_1.1.9.BIN -q"
Note that by setting a node's 'nextinstallmode' to NOSYNC, you are telling it to skip image re-synchronization on the next boot. After this boot, everything will be back to normal. It is probably wise to do the update on a small number of nodes first (e.g. 5-10) so that all the nodes are not bricked if something goes wrong. Starting cautiously by doing it on one node is probably a good idea.