Notes on upgrade of Lustre to 2.1.6 from 1.8.4

With the implementation of SL6.4 everywhere it became necessary to also upgrade Lustre from 1.8.4, which was not really compatible with the newer OS version. We chose to use the maintenance branch of Lustre that supports the SL6.4 kernel, which is Lustre version 2.1.6

Shutdown sequence of Lustre 1.8.4

• Stop all T3 batch processing so Worker Nodes would be idle in Lustre
  • On each do "service lustre_mount_umt3 stop"
  • Reboot/power off any machine with hung lustre processes
• Stop the mgs service on mgs.aglt2.org
  • umount the device
  • comment out in fstab
• Stop the mdt service on lmd02.aglt2.org
  • umount the device via service MountMDT stop
• Do the various, recommended backups of both the mgs and mdt file systems following the Lustre 1.8 manual
  • dd copy
    • dd if=/dev/sdb of=/mnt/lustre-backup/mdt_dd_backup.dd bs=1M
    • This took ~27hrs
  • tar and attribute copies
    • mount -t -o user_xattr -o acl ldiskfs /dev/sdb /mnt/tmp
    • cd /mnt/tmp
    • tar czf /root/mdt_bak.tgz --sparse .
    • getfattr -R -d -m '.*' -P . > /tmp/mdt_ea_new.bak
    • This took < 5hrs total
• Create a new volume of 500GB on the Sun NAS for the mdt, down from the 2TB volume originally used

root@lmd02 ~# mkfs.lustre --fsname umt3 --mgsnode=10.10.1.140@tcp0 --mdt --failnode=10.10.1.48@tcp0 \
   --failnode=10.10.1.49@tcp0 /dev/mapper/mpath4

   Permanent disk data:
Target:     umt3-MDTffff
Index:      unassigned
Lustre FS:  umt3
Mount type: ldiskfs
Flags:      0x71
              (MDT needs_index first_time update )
Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro
Parameters: mgsnode=10.10.1.140@tcp failover.node=10.10.1.48@tcp failover.node=10.10.1.49@tcp mdt.group_upcall=/usr/sbin/l_getgroups

checking for existing Lustre data: not found
device size = 512000MB
2 6 18
formatting backing filesystem ldiskfs on /dev/mapper/mpath4
        target name  umt3-MDTffff
        4k blocks     131072000
        options        -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups,mmp -F
mkfs_cmd = mke2fs -j -b 4096 -L umt3-MDTffff  -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups,mmp -F /dev/mapper/mpath4 131072000
Writing CONFIGS/mountdata

• Restore the mdt to the 500GB volume,
  • mount -t ldiskfs -o user_xattr -o acl /dev/mapper/mpath4 /mnt/tmp
  • cd /mnt/tmp
  • tar zxpf /root/mdt_bak.tgz --sparse
  • setfattr --restore=/tmp/mdt_ea_new.bak
  • rm OBJECTS/* CATALOGS

OST servers and NFS servers

• Copy a bunch of relevant files from each machine. See typical list below.
• Upon rebuilt, add modprobe.conf entry as modprobe.d/lustre.conf
• Make sure routing and networks are correct
• Re-install Lustre kernel and associated rpms

# Copy the saved line taken from /etc/modprobe.conf, back to /etc/modprobe.d/lustre.conf
# Typical values:
#    OST servers
# options lnet networks=tcp0(bond0) routes="tcp2 10.10.1.[50-52]@tcp0"
#    mdt servers
# options lnet networks=tcp0(bond0) routes="tcp2 10.10.1.[50-52]@tcp0"
#    mgs servers
# options lnet networks="tcp0(eth0),tcp2(eth2)"
#
# automount does not necessarily work correctly after CF3 runs until either a reboot, or...
#
   45  service rpcbind restart
   46  service rpcidmapd restart
   47  service autofs restart
#
   48  cd /atlas/data08/ball/admin/LustreSL6/server/
   50  rpm -Uvh kernel-2.6.32-358.11.1.el6_lustre.x86_64.rpm kernel-firmware-2.6.32-358.11.1.el6_lustre.x86_64.rpm kernel-headers-2.6.32-358.11.1.el6_lustre.x86_64.rpm
   51  reboot

# check to see that we are on the correct kernel, and if so, continue installing
  259  uname -r

2.6.32-358.11.1.el6_lustre.x86_64

  260  cd /atlas/data08/ball/admin/LustreSL6/server/
  261  yum localinstall \
e2fsprogs-1.42.7.wc1-7.el6.x86_64.rpm \
e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64.rpm \
libss-1.42.7.wc1-7.el6.x86_64.rpm \
libcom_err-1.42.7.wc1-7.el6.x86_64.rpm \
lustre-2.1.6-2.6.32_358.11.1.el6_lustre.x86_64.x86_64.rpm \
lustre-ldiskfs-3.3.0-2.6.32_358.11.1.el6_lustre.x86_64.x86_64.rpm \
lustre-modules-2.1.6-2.6.32_358.11.1.el6_lustre.x86_64.x86_64.rpm \
lustre-tests-2.1.6-2.6.32_358.11.1.el6_lustre.x86_64.x86_64.rpm

# typically installs 12 packages (dependencies) and updates 4

  262  reboot

• Add lustre volumes back to /etc/fstab
• Add mount points in /mnt as indicated by /etc/fstab lines
• mount -av
• Rerun "cf-agent -K" to pick up mount point monitors

-- BobBall - 11 Jul 2013