Research Computing Group Update: Condor1000 Project Harnesses Teraflops of Power for RIT
The following article is reprinted with permission of the Research Computing team about its recent news about increasing RITs research computing power and access. More information about Condor, in general, and the application at RIT can be found at: http://rc.rit.edu/systems.html and http://www.cs.wisc.edu/condor - ed.
On December 21st, 2007 Research Computing achieved its goal of harnessing the power of 1000 processors by tapping into the unused computing cycles of hundreds of computers at RIT. Actually, there are well over 1000 processing cores that are part of the RIT Condor pool and that number continues to grow day by day. Computing power is being utilized in the Center for Imaging Science, Computer Science, Research Computing, Information Technology Services, Information Technology, Math, Electrical Engineering, Mechanical Engineering, Manufacturing and Mechanical Engineering Technology, Liberal Arts, (and soon the Library, Imaging Arts and Sciences, and more).
Multiplying the power of a single desktop computer by 1000 or more is a tremendous increase in personal computing capacity and extends the boundaries of research problems that can be computed. Essentially, this is a campus computational grid. In total, the current Condor system consists of more than 1.5 terabytes of memory, greater than 67 terabytes of disk space, and about .75 teraflops of computing power.
What is Condor?
Condor is a project of the University of Wisconsin. It is what is known as High Throughput Computing, that is, the goal is to run as many jobs as possible at the same time. The easiest way to do this is to tap into all the computers on campus when they aren't being used. Condor senses when the computer is free according the policy on that particular machine (the policy is set by the owner of each machine. e.g., "only use for condor jobs between 11pm and 6am", etc.). The harvesting of these unused cycles ensures that the investment in all that computer hardware is utilized to the maximum extent possible.
Efficiency is not the only benefit from Condor. Researchers benefit by getting access to a huge number of computer processors at essentially little or no cost. Certain types of computer problems are ideal for the Condor environment. These jobs are ones that can be separated into hundreds or thousands of little jobs that run independently of one another but whose results can be aggregated at the end. An example of this is image processing or video. Some analysis requires the processing of each individual pixel of an image, or of a single frame of a movie. Each of these individual jobs can be run separately and the results brought together at the conclusion of the job.
Condor at RIT - Where did it come from?
The research staff and faculty in the Center for Imaging Science needed more processing power to handle images with high resolution and multi-spectral bands. Researchers in the Digital Imaging and Remote Sensing (DIRS) laboratory and the Laboratory for Imaging Algorithms and Systems (LIAS) used Condor software to meet these needs.
In 2006, Research Computing started to help support the Condor flock of workstations in Imaging Science. In the Spring of 2007, Condor was installed on the Research Computing cluster alongside other job managers and schedulers. By the Summer of 2007 the possibility of managing 1000 condor clients became an achievable goal. In the Fall of 2007, Research Computing began installing Condor on Macintosh computers in the Math Department. And by the end of 2007, over 1000 processors or cores were available on Windows, Macintosh, Solaris, and linux computers among all the flocks on campus spread across eight departments. Condor1000 was a reality.
...and where is it going? Grid Computing
With the success of the Condor1000 Project, RIT can not only support more researchers on campus who will utilize the system, but RIT can now begin to experiment with connecting its system with systems at other institutions to create pools of thousands and tens of thousands of processors in size. The advantage of such a large pool is threefold - larger jobs can be run, more users can run at the same time, and backup and redundancy are available, should there be any outages on campus. This is the benefit of a grid of computers: a shared cyberinfrastructure that increases efficiency, scope and productivity. RIT's membership in the statewide NYSGrid and the worldwide Open Science Grid will help enable such a scenario.
To read more about cyberinfrastructure initiatives across the county, and the role high performance computing can, and will, play at universities and research institutions, see the NSF document:
http://connect.educause.edu/Library/Abstract/CyberinfrastructureVision/37146?time=1199893261
Questions or comments? Email us at macits@rit.edu