The latest generation of storage purchased here uses Dell MD3260 and MD3060e storage shelves. This dense storage solution provides more than 600TB of storage in 18U of space. It was used as part of SuperComputing 2012 demonstrations and was used again during the OpenFlow demo in SuperComputing 2013
We have done some benchmarks on RAID6 and DDP configurations of this system. Our testing also included rebuild times after disk failure and performance during rebuild.
Our typical storage node consists of a Dell R710 + attached MD1200 disk units. This provides 180TB of storage in 14U of rack space.
US ATLAS Tier-2s have found this following configuration both powerful and inexpensive Individual storage nodes have exceeded 750MB/sec on the WAN.
At one time we tested configurations for this type of node very extensively. More information on these tests and other tests can be found on our wiki
Our site heavily utilizes an iSCSI SAN implemented as two vlans on our regular switching infrastructure. Our iSCSI infrastructure serves as storage for our VMware HA cluster, for our calibration database, and for our Lustre meta-data server.
Our latest generation of storage node consists of 2 MD3260 shelves each with an MD3060e expansion shelf. They are attached to an R720 headnode with 2xE5-2665 processors and 256GB of RAM. This is a very dense system and offers a new type of storage array configuration called Dynamic Disk Pools. We found that conventional RAID-6 had about double the speed when writing contigous blocks of data and was thus better suited to our needs (read speeds are similar). Our wiki has more information about benchmark testing.
This system was utilized for the first time for 100Gbps data transfer demonstrations during SuperComputing 2012 and was used again during the OpenFlow demo in SuperComputing 2013. R720 not pictured; external appearance is identical to R710 though it is internally improved. Most notable is the use of Gen 3 PCI-Express bus.
The R710 has two 8x PCIe slots and two 4x PCIe slots. We occupy three slots with H800 controllers and the last (8x) slot with a 10GB network interface card.
The MD1200 holds 12 disks and has 2 enclosure management modules so they can be cabled redundantly.
The storage for our VMware HA cluster is all accessed via an iSCSI SAN. It is a requirement for a highly available system such as ours that the storage be on some kind of shared storage mechanism so that all potential cluster physical machines can access virtual machine images. We use a Dell MD3600i for this storage. It has two 10GB 10GBase-T (copper RJ-45) ports on two redundant controllers.
This system also hosts one half of a redundant mirror managed by Oracle Advanced Storage Manager which holds our MDT calibration database.
The Sun/Oracle 7410 Unified Storage system consists of a headnode and two J4400 disk shelves with 24 disks per shelf. It also includes 2 SSD drives used for caching and filesystem journal logging. We use this system as an iSCSI device to host our Lustre meta-data volume and as one half of an Oracle ASM redundant mirror for the MDT calibration database. It has 8 x 1GB ports on the headnode which we aggregate into an 8Gb link for all VLANs required.
Our AFS network filesystem and host filesystems are backed up daily to a Sun/Oracle StorageTek SL500 tape library system. Our system holds 35 800GB LTO-4 tapes and is attached to a Sun/Oracle X4500 which provides 10TB of additional space used for disk-to-disk backup cache (image at right is representative of an X4500 as well). Our backups are managed by Amanda Backup and MySQL ZRM to backup databases.
Hiding in between the Dell nodes we have a Sun/Oracle X4540. This is a unique design with 48 top-loading disks. In our configuration, with 1TB disks, it provides 40TB of usable storage in 4U of space. We currently run FreeBSD on our X4540 and Solaris on our X4500 backup server. We also have a Sun/Oracle X4500 serving as our network backup host.
