Provisioning the Dell 2950/MD1000 Storage Servers

Recipe for getting a new PE2950 and MD1000 combination going as an fss-nfs appliance in our ROCKS cluster.

References

• pe2950 manuals http://support.dell.com/support/edocs/systems/pe2950/en/hom/html/index.htm
• md1000 manuals http://support.dell.com/support/edocs/systems/md1000/en/index.htm
• OMSA manuals http://support.dell.com/support/edocs/software/svradmin/
• DellServerProposedLabeling

Hardware Setup

The PE2950 has three adapter cards installed: 10GE in slot 1, PERC 5/E in slot 2, PERC 5/E in slot 3.

The two PERC 5/E cards give a total of 4 SAS connectors. Each is cabled to the left most connector on the back of one MD1000 shelf.

Connections

This is the layout for the storage nodes at MSU. Two shelves are mounted above and two below the pe2950 server. The PERC cards are in the PCI-E slots labeled 2 and 3. These are to the right (looking at rear of machine) with 2 on top and 3 on bottom. The PERC connectors are labeled 0 (left) and 1 (right). Upper PERC card servers upper shelves... PERC port 0 connects the upper of the pair of MD1000s. There are two interface to each MD1000, the left ones are in use and the rights are not. The cables are labeled A,B,C,D. The MD1000s are also labeled with A,B,C,D. Linux initializes the PERC card in slot 3 before slot 2. So the lower pair of MD1000s become /dev/sdb and the upper pair become /dev/sdc.

PCI Slots and Cards (note that PERC IDs and PCI Slot IDs are in reverse order...)

Note all positions are as viewed from REAR of chassis.

PCI Slot ID (on chassis / in omreport)	PCI ID (lspci)	Position	Card	Slot Type	Device Name RAID60
1	0c:00.0	Left (Central Riser in Manual)	Myricom 10GE	PCI-E x8	eth2
2	0e:00.0	Right Top (Left Riser in Manual)	PERC6/e ID 2	PCI-E x8	sdc
3	0a:00.0	Right Bottom (Left Riser in Manual)	PERC6/e ID 1	PCI-E x4 (x8 connector)	sdb

From PCI slot to enclosure ID...

What	Index	Position	Controller ID	Controller PCI Slot	Connector	Enclosure ID	Device Name
MD1000	A	Top	2	2	0	0:0	sdc
MD1000	B	Second to Top	2	2	1	1:0	sdc
PE2950	-	Middle
MD1000	C	Second to Bottom	1	3	0	0:0	sdb
MD1000	D	Bottom	1	3	1	1:0	sdb

MD1000 Layout

There is a simple script in /home/install/site-tools/md100-service-tags.sh that lists the service tags based on the above layout. Output is on page MsuStorageServerLayout.

Switch Setup

Add the VLAN config to the switch stack for the ports used by the PE2950. The 10GE connection goes on the public VLAN.

BIOS Setup

We want the PE2950 to network boot. I booted the node and set the first NIC to be first in the boot order.

At the same time, I recorded the MAC address of the first NIC.

ROCKS Database

I manually added the node to the ROCKS database (vs. using insert-ethers)

-bash-3.00# rocks dump | grep msufs02 | sort 
/opt/rocks/bin/rocks add host interface msufs02 eth0
/opt/rocks/bin/rocks add host interface msufs02 eth1
/opt/rocks/bin/rocks add host msufs02 cpus=8 rack=103 rank=14 membership="FS-server-nfs"
/opt/rocks/bin/rocks set host interface ip msufs02 eth0 10.10.2.22
/opt/rocks/bin/rocks set host interface ip msufs02 eth1 192.41.231.22
/opt/rocks/bin/rocks set host interface mac msufs02 eth0 00:1c:23:c1:a8:a2
/opt/rocks/bin/rocks set host interface mac msufs02 eth1 00:1c:23:c1:a8:a4
/opt/rocks/bin/rocks set host interface module msufs02 eth0 bnx2
/opt/rocks/bin/rocks set host interface module msufs02 eth1 bnx2
/opt/rocks/bin/rocks set host interface name msufs02 eth0 msufs02
/opt/rocks/bin/rocks set host interface name msufs02 eth1 msufs02.aglt2.org
/opt/rocks/bin/rocks set host interface subnet msufs02 eth0 private
/opt/rocks/bin/rocks set host interface subnet msufs02 eth1 public

(Actually ROCKS automatically added the eth1 MAC address as part of the install process.)

First Install

Boot the node and ROCKS will install it. Used SVN revision 77.

Trouble

The network didn't come up right, eth0 and eth2 were configured. eth2 wasn't passing packets (wouldn't ping and switch didn't have it's mac in address table). Cycling eth2 with ifdown/ifup got it going. Unexplained.

No default route set. /etc/sysconfig/network has:

root@msufs02 ~# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=msufs02.aglt2.org
NISDOMAIN=rocks
GATEWAY=192.41.231.1

Ok, interface is down again, and a default route can't be set when an interface is down apparently...

root@msufs02 ~# ifdown eth2
root@msufs02 ~# ifup eth2  
root@msufs02 ~# route add -net default gw 192.141.231.1 eth2
SIOCADDRT: Network is unreachable
root@msufs02 ~# ping 192.41.231.15
PING 192.41.231.15 (192.41.231.15) 56(84) bytes of data.
From 192.41.231.22 icmp_seq=0 Destination Host Unreachable
From 192.41.231.22 icmp_seq=1 Destination Host Unreachable
From 192.41.231.22 icmp_seq=2 Destination Host Unreachable

--- 192.41.231.15 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3010ms
, pipe 4

Ok, configured eth1, though didn't set it to automatically start.

Firmware

We want the BIOS, PERC 5/i and PERC 5/E firmware updated. The BIOS is in the Dell repository. The PERC firmwares will be applied manually.

Needed to install libsmbios rpms by hand, need to add these to ROCKS installer

Get the available firmware rpms installed to the node:

root@msufs02 ~# yum install `inventory_firmware -b`
SNIP

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 bmc_firmware_PowerEdge_2950  noarch     a06-18.1         fwupdate          244 k
 pci_firmware_ven_0x1028_dev_0x0015_subven_0x1028_subdev_0x1f01  noarch     a05-19.1         fwupdate          976 k
 pci_firmware_ven_0x1028_dev_0x0015_subven_0x1028_subdev_0x1f03  noarch     a05-19.1         fwupdate          976 k
 system_bios_PowerEdge_2950  noarch     2.1.0-18.1       fwupdate          429 k
Installing for dependencies:
 dell-bmcflash           x86_64     1.4.4-1.el4      dell-software     242 k
 dell-lsiflash           x86_64     1.4.9-1.el4      dell-software     1.6 M

Transaction Summary
=============================================================================
Install      6 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         
Total download size: 4.4 M
Is this ok [y/N]: y

SNIP

root@msufs02 ~# update_firmware --yes

Searching storage directory for available BIOS updates...

Checking Dell PERC 5/i Integrated Controller 1 Firmware - 5.1.1-0040
        Available: pci_firmware(ven_0x1028_dev_0x0015_subven_0x1028_subdev_0x1f03) - 5.1.1-0040
        Did not find a newer package to install that meets all installation checks.
Checking Baseboard Management Controller (BMC) for PowerEdge 2950 - 1.33
        Available: bmc_firmware(ven_0x1028_dev_0x01b2) - 1.77
        Found Update: bmc_firmware(ven_0x1028_dev_0x01b2) - 1.77
Checking Dell PERC 5/E Adapter RAID Controller 2 Firmware - 5.1.1-0040
        Available: pci_firmware(ven_0x1028_dev_0x0015_subven_0x1028_subdev_0x1f01) - 5.1.1-0040
        Did not find a newer package to install that meets all installation checks.
Checking System BIOS for PowerEdge 2950 - 1.5.1
        Available: system_bios(ven_0x1028_dev_0x01b2) - 2.1.0
        Found Update: system_bios(ven_0x1028_dev_0x01b2) - 2.1.0

Found firmware which needs to be updated.

Running updates...
BMC Upgrade disabled for now due to driver issues. It is being worked on.
 100% Installing bmc_firmware(ven_0x1028_dev_0x01b2) - 1.77
Done: Device update is disabled for this device.

 100% Installing system_bios(ven_0x1028_dev_0x01b2) - 2.1.0
Done: Device update was successful.

So, that procedure installed the system BIOS.

I've previously downloaded the PERC firmware .BIN files from dell.com...

root@msufs02 ~# omconfig system webserver action=stop
DSM SA Connection Service stopped successfully.

root@msufs02 ~# /etc/init.d/dataeng stop
Stopping Systems Management Data Engine:
Stopping dsm_sa_snmp32d:                                   [  OK  ]
Stopping dsm_sa_eventmgr32d:                               [  OK  ]
Stopping dsm_sa_datamgr32d: echo                           [  OK  ]

root@msufs02 ~# sh /home/install/firmware/dell/perc5i/FRMW_LX_R169302.BIN 
Collecting inventory...
..
Running validation...

PERC 5/i Integrated Controller 0

The version of this Update Package is newer than the currently installed
version.
Software application name: PERC 5/i Integrated Controller 0 Firmware
Package version: 5.2.1-0067
Installed version: 5.1.1-0040

Continue? Y/N:Y
Executing update...
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE
IS IN PROGRESS.
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!
................................
The operation was successful.
Would you like to reboot your system now?
Continue? Y/N:N
The system should be restarted for the update to take effect.

root@msufs02 ~# sh /home/install/firmware/dell/perc5e/FRMW_LX_R168186.BIN 
Collecting inventory...
..
Running validation...

PERC 5/E Adapter Controller 1

The version of this Update Package is newer than the currently installed
version.
Software application name: PERC 5/E Adapter Controller 1 Firmware
Package version: 5.2.1-0066
Installed version: 5.1.1-0040
PERC 5/E Adapter Controller 2

The version of this Update Package is newer than the currently installed
version.
Software application name: PERC 5/E Adapter Controller 2 Firmware
Package version: 5.2.1-0066
Installed version: 5.1.1-0040

Continue? Y/N:Y
Executing update...
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE
IS IN PROGRESS.
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!
...........................................................................
The operation was successful.
Would you like to reboot your system now?
Continue? Y/N:N
The system should be restarted for the update to take effect.

root@msufs02 ~# omreport storage controller
No controllers found

root@msufs02 ~# shutdown -r now

Setup RAID Arrays

Used scripted omconfig commands to setup RAID 50 arrays. After running the script, omreport shows the two arrays at /dev/sdc and /dev/sdd. These will be at /dev/sda and sdc after a reboot.

Install Again

Set the node to install and reboot.

Install Changes

I didn't put the nodes in /home/install/extras/fix_hostname.pl since their names where defined by hand to be what we want in the rocks db (vs. insert-ethers using fss-nfs-rack-rank).

I tried removing eth1 and adding eth2 in the ROCKS db. ROCKS install won't configure eth2, because the driver isn't available and the interface can't be brought up during the install. A work around to this issue is to add the proper module to modprobe.conf and output of dbreport ifcfg eth2 hostname to /etc/sysconfig/network-scripts/ifcfg-eth2. This can be done at install time in myricom.xml.

Creating the filesystems on MD1000s is done in an /etc/rc.d/rocksconfig.d script written from nfs-disks.xml.

Post-Install Config

Want to try using a process that invokes a single install which doesn't configure the MD1000s, with scripted configuration of the MD1000s and their filesystems as a post-install operation.

For now, since we haven't tested 9000 Byte frames at MSU, setting the 10GE interfaces to 1500 byte MTU (they default to 9000).

Formatting. The large XFS volumes are setup with external log partitions. These log partitions are defined in the set-partitions.sh script.

When vdisks are made on the MD1000s, the end up at sdc and sdd. After a reboot they are at sdb and sdc. Not sure if there is another way to get them moved other than rebooting...

-- TomRockwell - 09 Jan 2008