Studio for Scientific Imaging and Archiving of Cultural Heritage
A three-year grant from the Andrew W. Mellon Foundation has established the Studio for Scientific Imaging and Archiving of Cultural Heritage at RIT, a research, outreach, and service facility under the leadership of Dr. Roy S. Berns, R. S. Hunter Professor in Color Science, Appearance, and Technology within the Program of Color Science and Munsell Color Science Laboratory. The vision statement for the Studio is to promulgate scientific imaging practices within cultural heritage institutions. We firmly believe that the best way to effect lasting changes in how institutions digitize their cultural assets is to lead by example. The proposed Studio will contain aspects of both imaging services and conservation science departments found at museums, libraries, and archives. The Studio is a natural extension of previous research sponsored by the Andrew W. Mellon Foundation with Dr. Berns as principal investigator.
Outreach is a key component of the Studio, addressing the education barrier to incorporating scientific imaging within imaging services.
We will continue to submit papers for conferences and refereed journals, in similar fashion to past projects. There will be more emphasis on conservation than in the past, such as the American Institute of Conservation, the CIC Image Archiving conference, and the Museum Computer Network.
A short course will be developed to teach color science to artists, conservators, and curators. The will improve the general knowledge of museum personnel and provide important background that will aid in knowledge transfer of the results from the proposed project. It will also be a vehicle to demonstrate applications of spectral and four-light imaging systems. For example, a painting imaged with the four-light system was input to Maya, a computer-graphics rendering software package used by animators. A virtual museum gallery was produced and the painting rendered for spot illumination, shown in the figure above. Such images and walk-through animations will be included in the short course notes.
Workshops will be developed that are hands-on, in similar fashion to the workshop given during the previous Mellon-funded project, “Improving Artwork Reproduction Through 3D-Spectral Capture and Computer Graphics Rendering – Phase 2.” These will be presented both at RIT and museums.
Roy Berns will visit three institutions per year, on average, and demonstrate the techniques developed and tested at the Studio. Each visit will last approximately one week. Equipment will be brought as needed. During the visit, a workshop and short course will be delivered. Such visits will be extremely valuable as a source of knowledge transfer for both imaging services and Berns.
Another type of outreach is software. Any software developed during the proposed project will be downloadable for free from www.art-si.org. We are using the Matlab programming language and have purchased a commercial license during 2012 using Berns' discretionary funds. This enables executable software to be written and distributed, either as a commercial product or for free at our discretion. Funds from the proposed project will be used to maintain the commercial license during the course of the project. As an example, all the four-light imaging software developed during the last Mellon-funded project is available for downloading. The software is executable and includes a graphical interface to run the software. A screenshot from ArtViewer is shown in Figure 13, a program that renders images for user-selectable lighting geometry. The specific software to be written is listed in the expected outcomes.
Matlab was selected because we have over a decade of experience with this language and an extensive subroutine library of color-science tools.
The purpose of the Studio is to lead by example and prove that an imaging studio using commercial equipment can provide both images for science and scholarly communication. This approach will clarify the two imaging stages of analysis and synthesis. The archival images can be used directly by both conservators and conservation scientists. Following rendering (synthesis), images are used for printing and web display..
Four imaging systems are envisioned.
The first is easel based, using a high-resolution medium-format commercial camera and Xenon strobes, typically found in museum imaging services. Our particular camera is a new Sinar Photography AG system, consisting of a 86H 48MP color sensor, rePro camera body, eShutter with HR100 lens, and the CTM attachment. “CTM” stands for “color to match” the product name for a filter slider housing a pair of filters designed by the Roy Berns. When combined with software written at RIT, this system is capable of significantly higher color accuracy without a loss in spatial image quality. This is a commercialization of our Dual-RGB approach to scientific imaging. This approach was tested in our first Mellon-funded project. One feature of this camera is that by using only one of the filters, it produces images that are equivalent to a typical high-quality commercial camera. This system will have the greatest leverage towards demonstrating the advantages of scientific imaging within an imaging services studio. This system will image flat artwork and produce images appropriate for scientists, conservators, and photographers working in imaging services.
The second system will use the same easel and camera stand as the first system. A Canon commercial DSLR camera will be used. The system will be initiated using a Canon 5D Mark I and upgraded during the project to a higher-resolution DSLR. This will facilitate capturing data for computer graphics rendering of artwork using four-light polarization enhanced photometric stereo imaging, abbreviated as four-light imaging. This technique was developed during our last two Mellon-funded projects. This system will be used primarily for research since one of the proposed research themes will improve this system.
The third imaging system will be a “rapid system” using a Canon 5D Mark II DSLR modified for use as a Dual-RGB or standard DSLR. Rapid systems are gaining popularity for artwork that does not require the highest resolution and spatial image quality. The camera will be mounted on a motorized copy stand. A pair of Xenon linear strobes and controlling power pack will complete the system. This system will be used for imaging drawings, watercolors, photographic prints, and similar media for both research and imaging services. The first and third systems will be used for the imaging services. As such, we will not be imaging sculpture and other three-dimensional works.
The fourth system will be a spectral system, which consists of a 1.4MP Lumenera LW165m monochrome sensor (1392x1040 pixels) coupled with a CRI liquid crystal tunable filter. This system will be attached to the camera stand used in the first and second systems. The spectral system will be used primarily for research. Specifically, it will provide “ground truth” for our proposed research in improving pigment mapping. The output of this system will be a complete spectral reflectance curve for each pixel.
Rendering Scientific Images for the Web and Print
Bernie Lehmann,Apres Auvers, oil on canvas. Scientific image is on the left and display-preferred rendering is on the right. (The tone-mapping changes were amplified to clarify the type of adjustments necessary when rendering scientific images.)
Rendering scientific images addresses the assertion that scientific images are not appropriate for rendering. We will incorporate published literature on color and tone mapping that accounts for dynamic range, color gamut, and image size differences. Software will be developed that automatically re-renders image data that is based on user-supplied input such as output media and size. We will verify the algorithms with visual experiments where subjects will compare different approaches to rendering with the actual artwork. As an example, a painting was imaged using a scientific approach, shown above on the left. Since most displays have ambient flare (light reflecting off the display screen) and we view displays in fully lit rooms, a change in tone mapping is required for images rendered for web sites, shown above on the right.
A second component of this research theme is improving the scientific quality of existing camera systems in museums, archives, and libraries. Although cameras producing more than the usual RGB data can achieve the highest accuracy, significant improvement can be made with RGB cameras using profiling software optimized for scientific imaging. This software was developed in the last Mellon-funded project and is a component of the four-light imaging system. This piece of the software will be extracted, improved, and made into a stand-alone program that any camera system can use.
Improving Spectral Estimation of Using Commercial Camera Systems
The second research theme will demonstrate that conservators and conservation scientists can use images produced in an imaging services studio. In the past, we have used science to improve the color accuracy of images produced in imaging services. In this research, we will develop techniques to expand the utility of scientific images.
For conservators and conservation scientists, spectral data are more useful when such data are applied to pigment mapping, pigment selection for inpainting losses, and simulating proposed treatments, among others. The spectral data can be obtained using a number of different approaches including hyper-spectral (dozens to hundreds of channels sampling the visible and NIR spectral regions), multi-spectral (five to a dozen channels using a monochrome camera and either interference filters or narrow-band colored LEDs), or Dual-RGB. Despite these many approaches to spectral imaging, there is a paucity of software to process the images that is useful for art conservation. In this research, improvements will be made to our previous research in pigment mapping. Two approaches will be implemented. The first uses a priori knowledge such as the artist’s palette or direct spectral measurements using a small aperture spectrophotometer, a device quite common in imaging services, surprisingly. The second will combine low-resolution hyper-spectral and high-resolution multi-spectral or Dual-RGB images yielding high-resolution hyper-spectral images. This produces images of much greater resolution than achievable by cameras designed for spectral imaging. As an illustrative example, a painting was imaged using a Dual-RGB camera system producing the scientific image shown above. Assuming a known palette of paints and their optical properties, the image was mapped to concentration, and then used to generate a high-resolution spectral image.
Implementing a Spectral Color Management Workflow
The third theme will incorporate spectral color management and workflow into the proposed Studio imaging services. This research theme will provide further evidence of the utility of scientific imaging within imaging services and demonstrate that the Studio is at the forefront of color reproduction. One of the PI’s part-time doctoral students is a working member of the International Color Consortium (ICC) involved in spectral color management workflows. (All imaging services studios use an ICC workflow through Adobe Photoshop.) We will incorporate ICC advances in our imaging practices. We have already made progress in this area. Our current multi-spectral or Dual-RGB image encoding is a multi-channel tiff file where the first three channels are rendered for ProPhotoRGB and the remaining channels are the multi-spectral or Dual-RGB data, useful for scientific analysis, spectral color management, and spectral estimation. A screen-shot from Photoshop is shown in the above image.
Participating in spectral-based ICC activities will insure that new imaging systems and future versions of Adobe Photoshop will have built-in features for our approaches to changing imaging services practices. It will demonstrate that the Studio is leading the museum imaging community.
Improving Four-light Imaging
Diffuse image for a set of targets where the diffuse data were based on averaging four different lighting geometries.
The fourth theme addresses a lack of turnkey software and creates another scientific imaging service. In our last Mellon-funded project, we developed the four-light system that captures diffuse color and surface normal information. After using the system for the last six months, we have found that there are opportunities for improvement including spatial quality, incorporating conventional imaging, adding a metallic feature to the rendering software, and improving realism. For example, several panels from the Artist Materials Database were imaged using the four-light system, as seen above, Theoretically, there should not be any observable texture for these samples and the deployed software used the average of the four images to calculate the diffuse data. As seen below, the average method results in only a small amount of observable texture. However, because surface normal maps produce shading but not shadows, rendered images do not appear as realistic as conventional imaging. Furthermore, the images appear blurry because all the high-spatial-frequency specular data are filtered by cross polarization. By using the minimum of the four images rather than the average, the diffuse maps produce shadows, improving both image sharpness and realism. This is similar in concept to rendering the scientific images for display. An additional goal of this theme is increasing the utility of the image data, for example, documentation to quantify long-term color and surface changes.
Diffuse Image Comparison
Diffuse image comparison between average and minimum methods of combining four different lighting geometries.
Many past publications are available at the Art-Spectral Imaging site. Listed below are work for the current project, and moving forward from 2014.
GEOMETRY-INDEPENDENT TARGET-BASED CAMERA COLORIMETRIC CHARACTERIZATION
Farhad Moghareh Abed, Roy S. Berns, Kenichiro Masaoka
The Saunderson equations were used to compensate for differences in illuminant geometry between imaging systems and reference spectrophotometers. J. Imaging Science Technology 57, 050503-1 – 15 (2013). download article
WAVELENGTH-DEPENDENT SPATIAL CORRECTION AND SPECTRAL CALIBRATION OF A LIQUID CRYSTAL TUNABLE FILTER SYSTEM
Roy S. Berns, Brittany D. Cox, Farhad Moghareh Abed
A practical workflow was developed for the spectral calibration of a multispectral imaging system using liquid crystal tunable filter (LCTF) based . The workflow calibrated the system such the scale of spectral reflectance factor was transfered to the imaging system. The novel aspect of this research was a spatial correction accounting for the angular dependency of interference filters. Applied Optics, 54 3687-3693 (2015). download article
ARTIST PAINT SPECTRAL DATABASE
Roy S. Berns
Colorimetric and spectral gamuts are useful for the design of imaging systems such as display and print, to evaluate encoding errors for digital photography, for spectral reconstruction, and for lighting design. An artist paint spectral database was developed that includes spectra, colorimetry, eigenvectors, and optical data to produce additional spectra if desired. Nineteen acrylic-dispersion paints were sampled and Kubelka-Munk theory with the Saunderson correction was used to characterize each paint’s optical properties. Each chromatic paint or hue-adjacent paints were mixed computationally with white and with black at a range of concentrations producing approximately uniform sampling in CIELAB. In total, there are 23 hues and one gray scale with 770 unique spectra.IS&T Color and Imaging Conference, submitted (2016).download proceedings, download Excel spreadsheet
MODIFICATION OF CIEDE2000 FOR ASSESSING COLOR QUALITY OF IMAGE ARCHIVES
Roy S. Berns
Evaluating color accuracy of image archives is accomplished using test targets, colorimetric measurements, and total color difference calculations. Both CIELAB and CIEDE2000 are used with the later often preferred being an international standard and having a weighting function for chroma position. Its lightness weighting function is problematic when used for image archiving because it allows for greater lightness errors for dark colors, colors critical in defining image quality. A statistical analysis was performed on data used to derive CIEDE2000. The results do not support the weighting function for lightness. For imaging applications, it is recommended that the lightness weighting function be replaced with unity. IS&T Archiving Conference in press(2016). download proceedings
USING MAYA TO CREATE A VIRTUAL MUSEUM
Brittany D. Cox and Roy S. Berns
Surface normal and diffuse color maps were rendered for different lighting configurations using computer graphics software. The maps were measured using the four-light imaging system developed for the Studio for Scientific Imaging and Archiving of Cultural Heritage. IS&T Archiving Conference 51-55 (2015). download proceedings
IMAGING ARTWORK IN A STUDIO ENVIRONMENT FOR COMPUTER GRAPHICS RENDERING
Brittany D. Cox and Roy S. Berns
The four-light imaging system developed by T. Chen and Berns was improved using a heirarchical selection technique to remove specular highlights rather than cross-polarization. IS&T/SPIE Electronic Imaging 939803-939803 (2015). download proceedings
INCREASING THE VERSATILITY OF DIGITIZATIONS THROUGH POST-CAMERA FLAT-FIELDING
Joel Witwer and Roy S. Berns
When lighting to enhance surface texture, there is significant fall off. An image of this falloff using a white board can be used for archiving or not used to improve realism. IS&T Archiving Conference 110-113 (2015). download proceedings
ETRGB: AN ENCODING SPACE FOR ARTWORK IMAGING
Roy S. Berns and Maxim Derhak
A new encoding space, ETRGB (Extended Tristimulus RGB), was derived to facilitate accurate archiving of modern artist materials such as metallic flakes and interference pigments. IS&T Archiving Conference 74-77 (2015). download proceedings
Technical Reports SPECTRAL IMAGING USING A LIQUID CRYSTAL TUNABLE FILTER – PART II
Farhad Moghareh Abed and Roy S. Berns
A practical workflow was developed for the spectral calibration of a multispectral imaging system using liquid crystal tunable filter (LCTF) based . The workflow calibrated the system such the scale of spectral reflectance factor was transfered to the imaging system. The novel aspect of this research was a spatial correction accounting for the angular dependency of interference filters. (Refereed article below) download report
ARTIST PAINT TARGET (APT): A TOOL FOR VERIFYING CAMERA PERFORMANCE
Roy S. Berns
SPECTRAL AND COLOR CHARACTERISTICS OF BRONCOLOR PULSO F4 STROBE WITH UVE PROTECTION DOME
Roy S. Berns
Data were compiled for a Broncolor F4 strobe with UVE protection dome and CIE daylight spectral power distributions with equivalent correlated color temperature, distribution temperature, and 5500K. download report, auxiliary data
CAMERA ENCODING EVALUATION FOR IMAGE ARCHIVING OF CULTURAL HERITAGE
Roy S. Berns
A large set of colors were compiled including eight fluorescent artist paints, a sampling of glossy high-chroma artist paints, computationally extended artist paints, the Pointer gamut, and a computationally extended Pointer gamut. Their colorimetric data were used to evaluate potential encoding errors for sRGB, AdobeRGB(1998), ProPhotoRGB, and ProStarRGB. download report
A NEW ENCODING SYSTEM FOR IMAGE ARCHIVING OF CULTURAL HERITAGE: ETRGB
Roy S. Berns and Maxim Derhak
A new encoding space, ETRGB (Extended Tristimulus RGB), was developed to insure modern artist material such as those containing metals and interference pigments are encoded without errors caused by clipping or gamut mapping. (Proceeding below) download report, auxiliary data - ICC profile)
EVALUATING SOLID STATE AND TUNGSTEN-HALOGEN LIGHTING FOR IMAGING ARTWORK VIA COMPUTER SIMULATION
Roy S. Berns
Solid state lighting was evaluated for use as a taking illuminant in the imaging of artwork. Based on computer simulation of camera signals, a high color-rendering solid state light achieved the same accuracy as a Xenon strobe. download report
SPECTRAL SENSITIVITY AND TRANSMITTANCE MEASUREMENTS OF A SINAR 86H CTM DUAL-RGB DIGITAL CAMERA
Roy S. Berns and Yixuan Wang
The spectral sensitivity and spectral transmittance factor of the sensor and filters of a Sinar CTM (color to match) digital camera system were measured using a calibrated monochromatic source and spectrophotometer, respectively. download report, auxiliary data
GEOMETRIC DISTORTIONS IN A CANON EF 50MM F/2.5 MACRO LENS
LIGHTING EVALUATION FOR AESTHETIC IMAGING
The lighting capabilities of the Studio for Scientific Imaging and Archiving of Cultural Heritage were compared with typical practices at the Metropolitan Museum of Art. download report
INPAINTING SPREADSHEET OF GAMBLIN CONSERVATION COLORS
Roy S. Berns
Two-constant Kubelka-Munk theory with the Saunderson correction for refractive index discontinuity at the paint surface can be used to select pigments for inpainting leading to minimal metamerism. This Excel spreadsheet implements both colorimetric and spectral matching using the Gamblin Conservation Colors (circa 2000) as the optical database. The spreadsheet also includes a sheet to produce new optical data for your own paints. This is based on research by Berns and Mohammadi published in Studies in Conservation. download spreadsheet, download Studies article
Spectral Images of the Artist Paintings
Database of Spectral Images
The spectral information of a handful of paintings with known constituent pigments was extracted. Two major categories of pigments were selected: old master and moderns. Each category contains several paintings with different number of primary pigments indicated following table. A set of primary pigment that was used for creating the validation paintings is shown in the following image:
These set of pigments have been evaluated comperehensivley by Okumura. Full spectral and pigment information of his research can be downloaded here.
The spectral data has been extracted using a LCTF-based acquisition system explained here.
A number of the examined paintings are shown in the image below:
Spectral Images and metadata At this time spectral images are stored as matlab *.mat files. sRGB files are TIF. Click the thumbnails for larger versions. Absorption and scattering coefficients (.xlsx format) Dark current image (camera cap on) for noise characterization purposes. Flatfield image (from a uniform Lambertian flat surface) for noise characterization purposes.
This software is available for noncommercial, educational, scholarly, and/or charitable purposes only
Software in support of this research program is posted here. Most of this will be executable MatlabTM, which requires Matlab's compiler runtime. The specific version required is either R2014b (version 8.4) or R2016a (version 9.0.1). The 2014b software may not run on El Capitan MacOS.
These applications work on Mac OS. Running these applications may result in an error because they are from an unknown developer.
To bypass this, right click the application and select 'Open', then enter an administrator login.
X-rite ColorChecker Digital SG (CCSG) Evaluation
The CCSG is often used to build and evaluate camera color profiles. This software evaluates the colorimetric accuracy of the CCSG contained in an image. The zip file contains executable Matlab, instructions, reflectance data of CCSG, test image, example output, and source code.
MultiSpectralTools – Dual-RGB Processor
Collecting a pair of RGB images using different colored filters or lights can be used for spectral estimation and improving color accuracy compared with an RGB camera. This approach has been patented by RIT and commercialized by Flux Data and Sinar. Over a number of years, we have been improving our software. This is the March 2016 version. It runs on 16-bit linear Tiff and Sinar DNG file formats.
MultiSpectralTools has both colorimetric and spectral workflows. The file format is a nine-channel tiff, where the top RGB are color managed and the bottom six are the multi-spectral channels. These channels are used to display the average spectral reflectance factor of a user-defined circular aperture moved about the image. The colorimetric data, spectral data and a spectral reflectance factor plot can be saved if desired.
Four-Light Imaging Simple
Collecting images from four different lighting geometries enables calculating diffuse and surface normal maps. These maps can be used to render paintings and drawings for archiving and reprographics. When stored in .pfm format, our software ArtViewer can be used for real-time viewing. The program runs on several file formats including 16-bit linear Tiff and Sinar DNG.
Bit Depth Evaluation and Assignment
Sometimes following a conversion of a raw format to unassigned tif, the image appears very dark, a result of the sensor having a lower bit depth than 16 bit. This program evaluates and reassigns bit depth.
Convert CR2 to Tiff
Many CR2 to Tiff converters add a tone curve. This program maintains linearity. It is based on DCRAW (https://www.cybercom.net/~dcoffin/dcraw/). The zip file contains executable Matlab and source code.