SC Team at University of Michigan
Shawn McKee (AGLT2)
Roy Hockett (UM ITS)
Ben Meekhof (AGLT2)
During the SuperComputing 2012 (SC12) conference November 12-16, an international team of high energy physicists, computer scientists, and network engineers led by the California Institute of Technology (Caltech), the University of Victoria, and the University of Michigan, together with Brookhaven National Lab, Vanderbilt and other partners, smashed their previous records for data transfers using the latest generation of wide area network circuits.
With three 100 gigabit/sec (100 Gbps) circuits set up by Internet2, CENIC, CANARIE and BCnet, Starlight and US LHCNet, and servers at each of the sites with 40 gigabit Ethernet (40GE) interfaces, the team reached a record transfer rate of 336 Gbps between Caltech, the University of Victoria Computing Center in British Columbia, the University of Michigan, and the Salt Palace Convention Center in Utah.
This nearly doubled last year's overall record, and eclipsed the record for a bidirectional transfer on a single link with a data flow of 187 Gbps between Victoria and Salt Lake.
Press Release from Caltech
Press Release from ADVA Networking
Article at phys.org
Diagram of Umich and BNL site configuration on MiLR and Internet2 for SC12.
This plot from the UM Cacti system shows a large fraction of the 100 Gbps traffic on our link going to Starlight in Chicago which is labeled 'Internet2 Brocade MLX' in diagram (graph scale is Gigabytes per second so we see approximately 8 Gigabytes per second or 64 Gigabits per second).
The 100G wave terminated on a Juniper MX480 situated in the AGLT2 machine room. The MX480 additionally had a 4x40G blade and a 16x10G blade for use in connecting the local servers and networking infrastructure. AGLT2 connected its Dell/Force10 S4810 to the MX480 with 2 40Gbps links and 8 10Gbps links. Routing was enabled on the S4810 which allowed all of the production systems at AGLT2 to connect to the showroom floor.
In addition a new storage system based upon a Dell R720 headnode, 2 x MD3260 and 2 x MD3060e storage shelves was directly connected to the MX480 via 2 40G connections. The headnode had 2xE5-2665 processors, 256GB of RAM, a Mellanox dual ported 40G NIC (MCX314A-BCBT) and 4 SAS HBAs which connected to the 240 disks on the storage shelves. For SC12, 4 of AGLT2’s production dCache servers were configured to participate in the demos (each server had 2x10G connections into the S4810).
The MonaLisa Graph pictured here shows combined transfer rates to the SC12 floor for Caltech, Umich, Uvic, and to the Vanderbilt booth on the floor via Padtec. The combined rate is more than 375 Gbps
