Research Highlights / Full Story

RIT and HP

For nearly a decade, RIT and HP have engaged in sponsored research activities that support HP's product development goals. Recently, Dr. Eli Saber, an associate professor in the department of electrical engineering, has been leading an effort in collaboration with HP that focuses on developing and advancing novel methodologies for image segmentation. "By leveraging Saber's experience and expertise, we are able to address the needs in our color and imaging work," says Dr. Ken Lindblom, distinguished technologist at HP. "RIT is able to provide a unique skill set that augments what exists in our business division."

The interest in digital media has grown to new heights with the rapid technological advancements being made in the ability to capture and share digital images. This has necessitated the exploration of methods to interpret, organize, enhance, classify, and extract information from those images. Image segmentation is one approach that provides the foundation to make these functionalities more effective and expeditious.

Over the past two decades, many methodologies have been developed to tackle the complex problem of image segmentation, and the progression of these methodologies has resulted in the achievement of meaningful segmentations. However, the development of an effective simulated environment with realtime capabilities to perform this imaging task has proven to be extremely challenging. Saber, who is also an extended faculty member at the Chester F. Carlson Center for Imaging Science, along with Sreenath Rao Vantaram and Mustafa Jaber, imaging science doctoral students, and Mark Shaw and Ranjit Bhaskar, color and imaging R&D experts at HP, have developed a novel methodology that balances superior segmentation quality with speed. The patent-pending technique has been found extremely useful for diverse commercial and research initiatives related to the printing, medical, and remote sensing fields.

Novel Approach to Image Segmentation

The approach, Multiresolution Adaptive and Progressive Gradient-based color image SEGmentation (MAPGSEG), integrates color, texture, and gradient information in a multiresolution framework, and is based on the principle that the segmentation results of images at low resolution can be used to efficiently segment their corresponding high-resolution counterparts. The algorithm, which is implemented entirely in a MATLAB—an interactive numerical computing environment—is comprised of six modules.

The first module performs dyadic wavelet decomposition to obtain highquality approximations of the input RGB (red, green, blue color space) image at different resolution levels. The number of decomposition levels is automatically determined based on the smallest workable image dimension specified by a user or constrained by an application.

The second module converts the input image from RGB to the CIE L*a*b*, a device independent color space developed in 1976 by the Commission Internationale de l'Eclairage to achieve approximate perceptual consistency of colors. Consequently, the CIE L*a*b* color space facilitates improved color differentiation and separation of luminance-chrominance information, which enables the efficient handling of images with illumination variations. The color conversion scheme is followed by the computation of gradient information used to automatically and adaptively generate the thresholds required to process regions at different resolutions.

The third module performs a computationally efficient progressive and dynamic multiresolution region-growth procedure to create an initial estimate of regions in the image. In the fourth module, the MAPGSEG algorithm executes a texture characterization process, which differentiates textures at various resolutions. This module is an important aspect of the algorithm, as texture frequently manifests itself as multiple color shades or intensity variations due to material properties like density, gradient, and coarseness, impairing the process of image segmentation.

Next, the initially identified regions are integrated with the acquired texture model using a statistical region-merging procedure to obtain an interim segmentation. In the final phase, the acquired interim result is up-scaled and regions of high confidence are determined. These regions are passed on as a priori information to the next stage of the algorithm to process the remaining unsegmented areas with minimal computational costs. The algorithm progresses from one resolution level to another of the image pyramid until a segmentation result is achieved at the highest/original resolution and is semantically consistent with the input image.

Convincing Results

The team has evaluated the MAPGSEG technology using a prominent segmentation evaluation metric called the Normalized Probabilistic Rand (NPR) index on several hundred images made publicly available by the University of California at Berkeley. The results demonstrate their approach outperforms contemporary segmentation benchmarks with superior quality.

An example shown above illustrates a reasonably complex image of a church, which consists of stark illumination variations and a considerable number of structural details. By comparison, previous methods over-segment the image in the sky and dome regions due to illumination disparities. The structural details also cause these approaches to over-segment the facade of the church. The segmentation map that uses the novel MAPGSEG methodology illustrates the efficiency of the algorithm in handling issues of illumination. The images that proceed demonstrate the region growth strategy that merges similar color-texture regions independent of their spatial locations, resulting in an efficient and accurate image segmentation outcome.

The image of a Formula 1 racecourse presents a few challenges, such as the occlusion of image content by foreground objects as well as the presence of fine detail. Again, alternate methods over-segment the image, where MAPGSEG's region merging procedure successfully separates occluded regions and extracts fine details with great competence.

The segmentation task becomes extremely challenging when regions with dissimilar textures have great color similarity, such as the image of the cheetah. In this scenario, the texture descriptor is able to segment the skin tone that complements the background. The results shown emphasize the significance of the texture characterization module in the MAPGSEG approach.

"This research has catapulted us forward in our understanding of image segmentation," says Shaw. "We have an algorithm that has progressively gone faster, which we will be able use as a building block in our business."

Fostering Innovation

The research team has expanded on this work through the RIT Corporate R&D program. In partnership with Ortho Clinical Diagnostics, a Johnson & Johnson company, and Lenel, a UTC Fire and Security company, master's degree students are developing imaging algorithms specific to the industry partner's business needs.

The Corporate R&D program was pioneered by RIT's President Bill Destler to foster research partnerships among industry and universities. The agreement allows companies to retain the rights to any intellectual property that may be generated during the research projects, while RIT retains the right to publish and conduct further research that builds off of the work.

"The benefits are mutual," says Saber. "We are able to conduct applied research that has a direct implication to their business needs; meanwhile, our students gain tremendous confidence and become more than adequately prepared for a career in industry."

"Going forward, we view university partnerships as complementary to our internal development processes. Sometimes we have an idea for the future, but are unable to focus on it right away. That's the beauty of working with a university; we can tune the research to our specific needs and it becomes very beneficial to the business," explains Lindblom.

"While the private sector should not dictate the intellectual directions of the university, if universities and colleges are to become the economic engines they aspire to be, then the research and development activities need to be focused, at least in part, on projects that have the potential to lead to new products and services," adds Destler.