Supercomputer at RIT Takes on Black Holes and General Relativity

When black holes crash into each other at the center of a galaxy, the safest place to be is on the other side of the computer simulating the drama.

Scientists who study black holes simulate cataclysmic collisions on supercomputers that work around the clock churning out computations that would sizzle the latest desktop model.

Rochester Institute of Technology’s Center for Computational Relativity and Gravitation recently won $330,000 from the National Science Foundation to build a new computer cluster that will maintain the center’s competitive level of research in computational astrophysics and numerical relativity, a research field dedicated to proving Einstein’s theory of general relativity. The center’s research is relevant to such projects as the Laser Interferometer Gravitational Wave Observatory and the space-based Laser Interferometer Space Antenna, among others.

RIT scientist Manuela Campanelli leads the Center for Computational Relativity and Gravitation and the team that, in 2005, solved the 10 equations in Einstein’s theory of general relativity for strong field gravity—a discovery made possible through advances in computer technology and the team’s fresh approach to the problem.

Upon her arrival to RIT earlier this year, Campanelli and her team joined forces with those of physics professor David Merritt, who built his own supercomputer named “gravitySimulator”—a 32-node GRAPE (GRAvity PipEline) cluster for gravitational dynamics simulations.

Now, Campanelli’s team is building a new computer to remain at the forefront of their field. The computer, called “newHorizons,” will make the Center for Computational Relativity and Gravitation host to one of the largest computing facilities in the region.

“The new cluster will be the main work horse of the center,” says Campanelli, associate professor in RIT’s School for Mathematical Sciences. “It will give us the ability to do more refined simulations. It will also allow students to be able to work with us on projects.”

The computer cluster is a special-purpose machine designed with the best technology available, says Carlos Lousto, associate professor in the School for Mathematical Sciences.

“The kinds of computations we do are different, new,” says Lousto. “Before simulations can advance to the next level, all components of the computer must communicate.”

Lousto designed and built the computer using hardware from California-based Western Scientific. The 85 nodes that make this computer “super” each has its own dual processor, or four amounts of computing units per node. Direct communication between the nodes is made possible by AMD processors, allowing for high-speed interconnections called HTX or hyper thread connections.

Another unusual characteristic is that each node has 16 gigabytes of memory or a total of 1.4 terabytes of memory. In addition, infinite band technology makes the computer especially fast, moving “packages” of information with a lag time or latency of 2.9-microseconds—the fastest rate possible. The computer, which will have 36 terabytes of storage space, will operate at its maximum capacity 24 hours a day for four to five years.

Node-by-node, RIT’s new supercomputer outperforms the computers at the national labs, Lousto says.

“Other scientists have satellites and telescopes to do scientific research,” says Yosef Zlochower, a research assistant professor in the School for Mathematical Sciences. “We have supercomputers. It’s how we implement and test ideas. And because our simulations can take weeks, we needed the fastest machine possible.”

Computer scientist Hans-Peter Bischof and his students will also use newHorizon to visualize the results of the simulations. “A supercomputer like newHorizon is needed, in order to create useful visualizations out of terabytes of data,” says Bischof, associate professor in computer science at RIT.

The computer is kept in an air-conditioned room that never rises above 62 degrees Fahrenheit. The powerful, 20-ton air conditioning system cools the 85 nodes, each of which consumes 500 watts of electricity and generates a considerable amount of heat. Lousto designed the computer to maximize airflow and space between the clusters to prevent heat-related damage to the machine.

In addition, an automated alert system connected to a heat sensor will detect a rise in room temperature. And, if the electricity fails, powerful back-up batteries will keep the computer going for 15 minutes.

“The battery power will allow for a clean shut down of the machine without damage to hardware and loss of data,” Lousto says.

For more information about the Center for Computational Relativity and Gravitation, visit http://ccrg.rit.edu. The center is located in RIT’s School of Mathematical Sciences.

Rochester Institute of Technology is internationally recognized for academic leadership in computing, engineering, imaging technology, and fine and applied arts, in addition to unparalleled support services for students with hearing loss. More than 15,800 full- and part-time students are enrolled in RIT’s 340 career-oriented and professional programs, and its cooperative education program is one of the oldest and largest in the nation.

For nearly two decades, U.S. News & World Report has ranked RIT among the nation’s leading comprehensive universities. The Princeton Review features RIT in its 2007 Best 361 Colleges rankings and named the university one of America’s “Most Wired Campuses.” RIT is also featured in Barron’s Best Buys in Education.