Updating Dell Firmware

Dell provides (officially unsupported?) yum OMSA (OpenManage Server Administrator) repo. This includes the omreport/omconfig and other "srvadmin" tools. It now includes a usable system for doing firmware updates. The system takes the usual "DUP" (Dell Update Package) files and puts them in a yum driven framework. The system is able to determine which DUPs are needed on a system and apply them.

We have a mirror of the OMSA repo at http://mirror.msulocal/mirror/dell/hardware. A couple scripts for setting up that repo and running the firmware update are provided.

It doesn't seem practical to pull everything needed from this repo into ROCKS, so a post-install (might also perform pre-reinstall...) process is used.

The Tier3 ROCKS build is synchronized with the OMSA repo mirror, so that srvadmin* and related rpms don't get changed when the OMSA repo is bootstrapped.

Some hardware is not covered including the Intel x520 NICs.

Update Recipe

To update a node, we define the OMSA repo, and then run though a few simple commands. Note: the repo definition includes the node hardware type, there is a glitch with Scientific Linux worked around by making the /etc/redhat-release match RHEL systems, and the updates can take a long time to install.

On T3 worker nodes, installing two scripts using CFE. To setup the yum repo, do:

/root/tools/dell-omsa-setup-repo.sh

That script looks at the output of dmidecode to get the system type.

To bootstrap the framework and run an update (be patient, can take 10s of minutes):

/root/tools/dell-omsa-firmware-update.sh

The script will create /etc/redhat-release.sl and /etc/redhat-release.rhel. It tries to always put the proper .sl back in place (trap on exit). Note that this might fail in some instances. Also, the /etc/redhat-release file does get its timestamp changed (and may be broken for SELinux).

The script writes a log to /tmp/dell-omsa-firmware-YYYYMMDDHHMMSS.log

Also, the current repo server is not so robust, may have issues running large numbers of these at once. Should move the mirror to VMware cluster and possibly also use proxies for access.

Intel X520 10GE NICs

Works but needs DKMS rpm

Didn't work with UL kernel 3.2.32-UL1.el6.x86_64

The update process will work for the Intel X520 NICs, but dkms needs to be installed on the system and it is not included in OMSA repo (or SL6). It is in EPEL.

If DKMS is missing, will see a message like this:

# inventory_firmware
Wait while we inventory system:
Invalid XML from module /usr/libexec/dell_dup/dell_ie_nic_intel-1.1.0

Manual DKMS rpm install:

</verbatim> wget http://mirror.msulocal/mirror/epel/6/x86_64/dkms-2.2.0.3-2.el6.noarch.rpm yum localinstall dkms-2.2.0.3-2.el6.noarch.rpm </verbatim>

"iqvlinux" is the package being built, this is an Intel driver for interfacing to the NIC (me thinks). The next run of "inventory_firmware" will take extra time as dkms builds modules.

See http://lists.us.dell.com/pipermail/linux-poweredge/2012-August/046880.html Confusingly, the above worked even though the test system does not have gcc, and only has regular kernel rpm and not the kernel-devel or kernel-headers.

Changing to new version

Minimal method:

• point yum config to new location
• run yum update
• fiddle the redhat-release "cp /etc/redhat-release.rhel /etc/redhat-release"
• run update_firmware --yes

See http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/43136 for a description of complete removal / reinstall.

Issues

Have not yet seen issues with running production tasks on nodes that have the firmware installer installed, but have not done that widely yet. Expecting it to be OK though. Seems good to not mix versions of srvadmin though, hence the synchronization with ROCKS.

If the DRAC gives trouble, may need to disconnect power from node in order to restart DRAC.

Have had issues getting the repo mirrored successfully. To help with this, now mirror the repo twice and a "diff" can be done between the two copies. See the "do-mirror.sh" script in the mirror/dell directory.

Some of these installs take several minutes, for BIOS 6.3.0 on r610, setup of the BIOS package took 15 minutes and the reboot to apply the firmware took another 10, but it did work.

Maybe it makes sense to split the helper script up so that there is a separate script to run the firmware_update.

Update recipe

## Need to update tools, otherwise it just sees the bios.
wget -q -O - http://10.10.128.13/mirror/dell/hardware/OMSA_7.2.1/bootstrap.cgi | bash
yum install dell_ft_install

yum install $(bootstrap_firmware)
yum update
update_firmware --yes

# restart srvadmin-services.sh
srvadmin-services.sh start

# May also need to temporarily modify /etc/redhat-release to look like RHEL system if it complains about being unable to determine system generation

# firmware updates may require a reboot

Todo

• Update top instructions, merge with Update Recipe section

-- TomRockwell - 06 Nov 2012