Leading Physics Researchers Speed Up Data Movement Across Multiple Supercomputing Centers by Over 50x

 

Scientists at the MIMD Lattice Computation (MILC) Research Collaboration are using Globus in an important study that seeks to achieve a deeper understanding of the fundamental laws of physics. The study examines the validity of the “Standard Model of High Energy Physics,” the current standard for classifying subatomic particles, and also searches for new physics beyond it. MILC scientists are focusing on the quantum chromodynamics (QCD) field of the Standard Model, the theory of the strong interactions of subatomic physics, which promises to supply some of the most interesting predictions related to Standard Model validity.

To carry out the required QCD calculations, MILC scientists must perform large-scale numerical simulations where QCD is reformulated on a lattice, or grid. These simulations require a large amount of computing power; scientists end up performing segments of the study on a number of different supercomputers where time has been allocated (such as TeraGrid machines like Kraken, Athena, Lincoln and Longhorn; or DOE computers like Franklin, Intrepid and Hopper).  The use of these facilities ends up being not only costly but time consuming, since scientists must distribute their work to wherever computing time happens to be available.

With the frequent need to move very large files around, MILC scientists turned to Globus to simplify this process. According to Steven Gottlieb, Distinguished Professor, Indiana University and a senior member of the MILC Collaboration, the resulting speed up has been “very impressive,” with one hundred 7-gigabyte files transferred in just 90 minutes (a process that would have take over 3 days with scp).

MILC scientists also praised Globus’s convenience and ease of use – as Gottlieb put it, Globus “frees up my time to do more creative work rather than typing scp commands or devising scripts to initiate and monitor progress to move many files.” The reduction in delays and manual intervention has “made a big difference” in the convenience of moving projects between centers.

For all of his early use of the Globus system, including multiple large file transfers, Gottlieb was named Globus 'User of the Month' for June 2011. Click here to read the announcement.

Highlights:

• “Globus is the most beneficial grid technology I have even seen.”
• “I moved 100 7.3 GB files in about 1.5 hours. I am very impressed. I also like the new commands and help system.”
• “We transferred files at a rate of 1100 Mb/sec with Globus. The same transfer would have taken over 3 days with scp at a rate of about 20 Mb/sec.”
• “Globus has made a big difference in convenience – it’s now much easier to move the files we need where we need them.”
• "Globus has more parallelism and is faster.”
• “I have been able to transfer groups of files with relative ease.”
• “It is really convenient not to have to pre-stage files from mass storage. In my previous approach, copies would sometime fail because the data was not pre-staged -- I had to have a second script to check each file and try again. It was quite an annoyance.”