Simulated black hole event in ATLAS

ATLAS Computing and Muon Calibration Center

Enabling Scientific Discoveries with LHC Data Distribution Over 100 Gigabit Networks

SC Team at University of Michigan
Shawn McKee (AGLT2)
Roy Hockett (UM ITS)
Ben Meekhof (AGLT2)

During the SuperComputing 2012 (SC12) conference November 12-16, an international team of high energy physicists, computer scientists, and network engineers led by the California Institute of Technology (Caltech), the University of Victoria, and the University of Michigan, together with Brookhaven National Lab, Vanderbilt and other partners, smashed their previous records for data transfers using the latest generation of wide area network circuits.

Supercomputing Data Distribution

With three 100 gigabit/sec (100 Gbps) circuits set up by Internet2, CENIC, CANARIE and BCnet, Starlight and US LHCNet, and servers at each of the sites with 40 gigabit Ethernet (40GE) interfaces, the team reached a record transfer rate of 336 Gbps between Caltech, the University of Victoria Computing Center in British Columbia, the University of Michigan, and the Salt Palace Convention Center in Utah.

This nearly doubled last year's overall record, and eclipsed the record for a bidirectional transfer on a single link with a data flow of 187 Gbps between Victoria and Salt Lake.

Other News about our Demo...

Press Release from Caltech

Press Release from ADVA Networking

Article at

Other SC News...

HPCWire Article

CNET Article

Site Configurations for SC12 Collaboration

Diagram of Umich and BNL site configuration on MiLR and Internet2 for SC12.

Supercomputing Site Diagram

This plot from the UM Cacti system shows a large fraction of the 100 Gbps traffic on our link going to Starlight in Chicago which is labeled 'Internet2 Brocade MLX' in diagram (graph scale is Gigabytes per second so we see approximately 8 Gigabytes per second or 64 Gigabits per second).

Traffic Graph - Starlight link

SC12 Storage Setup at University of Michigan

The University of Michigan, partnering with ADVA and Juniper, created a 100 Gbps path between the ATLAS Great Lakes Tier-2 (AGLT2) Center on the Ann Arbor UM camps and the Internet2 Brocade MLX in Chicago. Internet2 then provided transport between Chicago and the SC12 showroom floor in Salt Lake City. AGLT2 provided the local networking and servers participating in SC12.

SuperComputing Network Diagram

The 100G wave terminated on a Juniper MX480 situated in the AGLT2 machine room. The MX480 additionally had a 4x40G blade and a 16x10G blade for use in connecting the local servers and networking infrastructure. AGLT2 connected its Dell/Force10 S4810 to the MX480 with 2 40Gbps links and 8 10Gbps links. Routing was enabled on the S4810 which allowed all of the production systems at AGLT2 to connect to the showroom floor.

In addition a new storage system based upon a Dell R720 headnode, 2 x MD3260 and 2 x MD3060e storage shelves was directly connected to the MX480 via 2 40G connections. The headnode had 2xE5-2665 processors, 256GB of RAM, a Mellanox dual ported 40G NIC (MCX314A-BCBT) and 4 SAS HBAs which connected to the 240 disks on the storage shelves. For SC12, 4 of AGLT2’s production dCache servers were configured to participate in the demos (each server had 2x10G connections into the S4810).

MonaLisa Graph of Network Transfers to SC

Network Transfers to SC12

The MonaLisa Graph pictured here shows combined transfer rates to the SC12 floor for Caltech, Umich, Uvic, and to the Vanderbilt booth on the floor via Padtec. The combined rate is more than 375 Gbps