Reflectance imaging spectroscopy in heritage science

Abstract

The present paper focuses on the reflectance spectral imaging of painted surfaces in the visible-near infrared spectral region (400–2500 nm). Other spectral ranges and methods are mentioned, to contextualize the spectral investigation of works of art.

1 Introduction

Imaging Spectroscopy (IS) is the acquisition of images in many contiguous spectral bands utilizing either reflectance (Reflectance Imaging Spectroscopy—RIS) or luminescence, based respectively on the diffusion or spontaneous emission of light [1]. In a more widely accepted sense, IS is the study of the interaction between matter and electromagnetic radiation (study of the absorption and emission of radiation by matter the wavelength of the radiation), displayed point by point as an image. Later, the concept was expanded greatly to include any interaction with radiative energy as a function of its wavelength or frequency, predominantly in the electromagnetic spectrum, though matter waves and acoustic waves can also be considered forms of radiative energy [2]. More recently, the definition has been expanded to include the study of the interactions between particles such as electrons, protons, and ions, as well as their interaction with other particles as a function of their collision energy. Newly, even gravitational waves have been associated with a spectral signature in the context of the Laser Interferometer Gravitational-Wave Observatory (LIGO) and laser interferometry. Spectroscopy, primarily in the electromagnetic spectrum, is a fundamental exploratory tool in the fields of physics, chemistry, and astronomy, allowing the composition, physical structure and electronic structure of matter to be investigated at atomic-, molecular-, macro- scale, and over astronomical distances. Important applications arise from biomedical spectroscopy in the areas of tissue analysis and medical imaging. In the remote sensing community, IS has many names including imaging spectrometry, hyperspectral and ultraspectral imaging [3], even though Ball in his book Spectroscopy (1995) argues that spectrometry be limited to measurement not including photons, as in mass spectrometry [4], but definitively claimed by Clark as appropriate [3].

The prefix “spectro-” derives from the Latin word spectrum, which means “image” or “apparition”, including the meaning “spectre” (ghost), and it is used to form words relating to spectra. Its meaning “visible band showing the successive colours, formed from a beam of light passed through a prism” is first recorded in the 1670s by Newton. Figurative sense of “entire range of something” is from the beginning of the 1990s. The suffix “-scopy” originates from Greek, i.e. “analysis, observation, examination”. A spectrometer is, then, a device used to record spectra and spectroscopy is the use of a spectrometer for chemical analysis.

Terminology and definitions evolve over time. RIS, multiband, multispectral and hyperspectral imaging (resp. MBI, MSI and HSI) can carry a range of definitions depending essentially on the application. Reflectance refers to the light reflected or scattered by a material relative to the incident light and the reflectance spectrum is a curve illustrating the amount of reflectance at each wavelength over a defined spectral range [5].

Fig. 1

Multiband imaging with bandpass filters (FWHM $> 100$ nm)

MBI (Fig. 1) generally refers to the acquisition of uncalibrated images with very broad bandwidths ($>100$ nm) captured using a CCD camera (e.g. a modified digital single lens reflex - SLR camera) and bandpass filters. In its basic definition, a colour image itself can be considered as a MB image, being the composition of a red (R), green (G) and blue (B) image (meaning images obtained with broad-band filters selecting the red, green, blue light, respectively). MBI captures characteristic spectral information about objects, however, the uncalibrated image set cannot produce reflectance spectra. Both MSI (Fig. 2) and HSI (Fig. 3) are considered RIS, defined as the collection of images at many different wavelengths to obtain reflectance spectra over a large spatial area [6]. In specific, Ricciardi et al. [6] define MSI as the acquisition of calibrated images with bandwidths of tens to hundreds of nanometers and HSI with bandwidths of a few nanometers or less. The number of bands is inversely proportional to their full width at half maximum (FWHM) (i.e. the smaller the FWHM, the higher the number of bands).

Fig. 2

Multispectral imaging (10 nm < FWHM $< 100$ nm)

Fig. 3

Hyperspectral imaging (FWHM $< 10$ nm)

In this respect, the term MBI, MSI or HSI describes a system capable to acquire respectively a few broad bands (FWHM $>100$ nm), a few tens of medium-width bands (10–100 nm FWHM) bands or hundreds/thousands narrow bands (FWHM $< 10$ nm). [7] Whatever definition is adopted, the basic concept underlying RIS is the acquisition of a stack of images generated by measuring the radiation backscattered by a surface when illuminated with a broadband source. Each image is characterized by a wavelength (an energy) interval whose way of selection defines the typology of instrument used. Even though only HS devices can be properly considered imaging spectrometers, MS instruments perform a low spectral resolution spectrometry, therefore fall in the category of imaging spectrometers.

The result of a RIS measurement is the so-called datacube or spectral cube (the adjective multi- or hyper- is often added, resulting in MS or HS cube), which contains one spectral reflectance image for each wavelength band. It can be visualized as a three-dimensional (3D) set of data composed of stacked images of the same scene seen at adjacent wavelengths, containing, thus, two spatial (coordinates x and y) and one spectral (wavelength l) dimensions. Therefore, its interpretation is two-fold:

• 1. reflectance imaging, i.e. each wavelength channel provides a monochromatic two-dimensional (2D) image;

• 2. reflectance spectroscopy, i.e. the pixel values at the same pixel coordinate in all images of the datacube correspond to an entire reflectance spectrum of the object at the corresponding location.

2 Spectroscopy: from lab to remote sensing

The history of spectroscopy, intended as the study of the interaction between matter and electromagnetic radiation, originated through the study of visible light dispersed into its wavelength. Its origin traces back to the optics experiments of Sir Isaac Newton who, in 1665, demonstrated that white light could be split up into component colours by a prism. Newton applied the word “spectrum” to describe the rainbow of colours that combine to form white light and that are revealed when the white light is passed through a prism: this latter phenomenon is at the basis of the optomechanical hardware of a spectrometer, thus marking the beginning of the history of spectroscopy. In 1800, Sir Frederick William Herschel, the German-born British astronomer, pioneered the use of astronomical spectrophotometry using a prism and a temperature-measuring equipment to measure the individual temperatures of each colour of the spectrum obtained from the wavelength distribution of stellar spectra. In the course of these investigations, Herschel discovered IR radiation: to his surprise, he found the highest temperature of all in this new spectral region where no sunlight was visible. New designs in optics, specifically diffraction grating, enabled systematic observations of the solar spectrum. During the early 1800s, Joseph von Fraunhofer conducted experiments with dispersive spectrometers that enabled spectroscopy to become a more precise and quantitative scientific technique, leading, in 1814, to the discovery of the dark absorption lines in the spectrum of the sun, now known as Fraunhofer lines. Since then, spectroscopy has played a significant role in chemistry, physics and astronomy. The subsequent history of spectroscopy provided the empirical foundations for atomic and molecular physics [8], whereas in modern physics, the discipline of studying light and interaction of light with matter is called optics [9].

For decades, spectroscopy has been used mainly in the laboratory to identify, understand, and quantify solid, liquid, or gaseous materials, based on the detection of absorption features due to specific chemical bonds. [10, 11] Spectroscopic measurements have a long history in the study of the Earth and planets for remote sensing to spectrally identify and spatially map materials. [1, 3, 12] Up to the 1990s, MSI instruments/experiments collecting high quality images in a few, broad spectral bands have dominated in this field. [13] The technological progress led to a new generation of sensors that made it possible to combine imaging with spectroscopy to create the new discipline of IS [14] in its modern acceptation. This novel generation of imaging spectrometers allowed to acquire data with enough spectral range, resolution, and sampling at every pixel in a raster image so that individual absorption features could be identified and spatially mapped (direct identification of surface materials on a picture-element basis accomplished by proper sampling of absorption features in the reflectance spectrum). The ability to acquire laboratory-like spectra remotely is a major advance in remote sensing capability. Concomitant advances in computer technology for the reduction and storage of such potentially massive data sets are at hand, and new analytic techniques are being developed to extract the full information content of the data. Remote sensing evolved almost half a century ago, mainly with the intention of distant terrestrial measurements and mapping of the Earth with space imagery, which was only marginally related to spectroscopic studies due to the initial very poor spectral resolution [15,16,17]. During the past three decades, the technological and computational advancements in the process of remote imaging procured with fine wavelength resolution [18,19,20,21], providing in situ spectroscopy from space to obtain information on the atomic and molecular compositions of terrestrial materials. Significant achievements in IS are attributed to airborne instruments, particularly arising in the early 1980s and 1990s [1, 22]. However, it was not until 1999 that the first imaging spectrometer was launched in space, the NASA Moderate-resolution Imaging Spectroradiometer (MODIS) [23]

The remote observation of the Earth resulted, then, in the development of new technologies that revolutionized the field of remote sensing. In the 90s, the first commercial spectrometers appeared on the market, making their use other than for aerospatial remote sensing possible: this was the kick-start for spreading IS out to other fields, such as pharmaceutical [24], agri-food [25], biology [26], medicine [27], chemistry [28], industrial sorting [29], quality control [30], surveillance [31], geology [3], forensic science [32, 33]. Once it became a well-established technique, its use in the heritage science (HS) turned out possible, however, it is not until 2000 that the first application in a museum appeared for the analysis, documentation and diagnosis of paintings [34, 35]. The peculiarity, unicity, frailness of works of art together with the necessity of having movable instruments to reach museums, churches and conservation centres hampered IS application of painted surfaces and brought out the need to redesign the instruments both to safeguard the objects during measurements and to ensure portability [36,37,38].

2.1 Development of imaging spectroscopy in heritage science

The driving force that triggered the research on IS applied in heritage science has been the desire to extend the potential of infrared reflectography (IRR), a consolidated technique for the diagnostics of paintings for the identification of details underlying the pictorial layer and not visible to the naked eye. Van Asperen De Boer [39] laid its theoretical and experimental bases in the wide-band modality, introducing the use of PbS Vidicon cameras, characterized by a broad spectral sensitivity (up to 2 micron). IRR consists in irradiating the painting/surface with an IR source, and in detecting the back-scattered radiation with a suitable device. Thanks to the transparency properties of most pigments to the IR radiation, it allows the visualization of features underneath the painting’s surface, such as the underdrawing (a preliminary sketch made by the painter on a preparation ground, prior to painting), the pentimenti (an underlying image providing evidence of revision by the artist), subsequent retouchings or overpaintings (generally speaking, foreign materials added at a later stage for modifying the artwork’s painted surface or structure), restoration intervention (the process of re-establishing the artwork legibility through selective removal of patina, consolidation of ancient materials and eventual reconstruction of missing pieces).

In the 80s, studies carried out at the National Gallery (NG) in London have implemented a colorimeter coupled to standard photographic imaging to detect chromatic changes. The first analogic broad-band imaging spectroscope was then realized with a set of RGB filters in front of the camera lens, and a further progress was reached by acquiring also the IR image. With the advent of CCD cameras, the very first attempt of imaging spectrometer was a 4-band device (RGB and IR) developed in 1988, once more at the NG [40].

In the early 1990s, a low-resolution IS ($< 10$ spectral bands) was first applied in situ to image paintings at the NG in London [41, 42]. Initially used for qualitative comparison between bands and to improve image colour accuracy, it became properly an IS technique for pigment identification in the late 1990s [43,44,45,46].

The research supported by several EU funded projects focused, then, on colorimetric analysis for accurate colour reproduction and for monitoring chromatic variations due to aging. The pioneer prototype VASARI, homonymous of the EU VASARI project (Visual art systems for archiving and retrieval of images, FP2-ESPRIT 2, 1989–1992) aimed at colour measurement, colour fidelity reproduction, and damage assessment [47]. The resulting two MS prototypes, providing 7 bands in 400-700 nm range, are kept at the NG (London) and the Döerner Institute (Munich), respectively. Within the subsequent EU MARC project (Methodology for Arts Reproduction in Colour, FP3-ESPRIT 3, 1992–1995), an integrated system for the digital acquisition, storage and handling of high-definition images of paintings for printing purposes was built and used to produce high-quality catalogues (not a spectral-based system). Driven by the need to increase public awareness of art images as well as by business opportunities appertaining to the printing of art images and building on VASARI’s legacy, the MARC project produced, among others, a catalogue showcasing paintings from different European museums to promote the established methodology. [48, 49]

In the same period, the EU MUSA project (Use of Multimedia for Protecting Europe’s Cultural Heritage, FP3-ESPRIT 3, 1992–1994) fostered the development of new software tools for the conservation and dissemination of works of art, based on high-quality images and data. [50]

The last endeavour in the series, the CRISATEL project (Conservation Restoration Innovation Systems for image capture and digital Archiving to enhance Training, Education and lifelong Learning, FP5-IST, 2001–2005) developed technologies to preserve paintings by ultra-high quality MS digitisation, enabling the use of the images as surrogates to assist conservation of the originals. Two systems were devised in 400–1000 nm range: the laboratory version in use at the NG and the JumboScan camera. The former is based on an off-the-shelf monochrome digital camera (1300 $\times$ 1030 pixel, pixel size 6.7 micron), mounted on an X–Y scanning stage for micro scanning mode (3900 $\times$ 3900 pixel), fitted with 13 band-pass filters (10 in VIS, FWHM=40 nm; 3 in NIR, FWHM=100 nm). The camera is refocused automatically with each change of filter because of filter’s thickness variation. The 80 mm lens focal length results in a 20 pixels/mm resolution on the painting at the closest object distance [51]. The latter MS scanner, the JumboScan, is based on a 12000 pixel linear array mounted vertically and mechanically displaced to scan 30000 horizontal positions (images up to 12000 $\times$ 30000 pixels). The sensor is equipped with the same filter wheel as the NG device [52, 53]. Both in Europe and in the US, CRISATEL prototype examined a variety of famous paintings such as the Mona Lisa [54, 55] and the Van Gogh’s Self-portrait at the NG of Art, Washington D. C. [56], paving the way for IS in the heritage science.

Yet in early 2000s, the achieved spectral information was not completely satisfactory, either for digital archiving or for colorimetric measurements, fostering the research towards higher spectral resolution. The pioneer study by Baronti et al. [43, 44] proposed a MS system based on a Vidicon camera equipped with a set of interferential filters on a rotating wheel (420–1550 nm spectral range). Bacci et al. [57] report on a PbS Vidicon camera system equipped with 32 interferential filters covering the 400–1700 nm spectral range at 10 nm spectral resolution. Based on point-wise detection, a novel scanning device was developed in the 380–800 nm spectral range. [58] The detector, a customized linear multi-anode photomultiplier, was composed of 32 elements, equipped with 10 nm wide interferential filters of 10 or 20 nm spacing. The image was captured by scanning the painting surface point-by-point, and MS acquisition was performed through a standard multimode fiber (200/250 micron core/clad diameter, NA = 0.22) coupled with a catoptric adapter, made of parabolic and cylindrical mirrors, which shaped a rectangular beam on the photomultiplier [58]. The authors further extended the spectral range by adding a separate IR module and acquiring the raster image by scanning the surface with a 4-by-4 square-shaped fiber bundle, coupled to a set of InGaAs photodiodes preceded by interferential filters [59, 60]. The MS scanner underwent multiple improvements in the following years [61,62,63] reaching its present 32 bands arrangement and up to 2500 nm spectral coverage [64,65,66].

The research on MBI [67] and MSI [68,69,70] proliferated, producing a flourishing method for relevant applications on cultural heritage objects [46, 57, 71, 72]. Delaney et al. [68] first extended the spectral range up to 2500 nm, whereas Mansfield et al. [73] introduced the use of tunable filters.

Parallel to the development of these MB and MS devices, the characterization of constituent materials and degradation products in artworks was carried out by fibre optic reflectance spectroscopy (FORS), demonstrating its power as non-invasive diagnostic tool [73,74,75,76,77,78,79,80] with relevant spectral database [81]. In this respect, the HSI can be seen as the two-dimensional (2D) implementation of FORS. [82,83,84]

Many instruments have been projected by different research groups all over the world. To name just a few, Casini et al. [35, 85] proposed a HS camera operating in the 400–900 nm spectral range coupled with a prism-grating-prism line-spectrograph resulting in a 500 bands device (FWHM = 1 nm) and 100 micron spatial sampling. The system was implemented with an InGaAs camera to cover also 900-1700 nm range [36, 37, 86]. Another Italian group set up a device for colour measurements based on the commercial spectrograph Imspector, working in 400–730 nm spectral range [87]. Balas et al. [34] proposed in 2003 a HS device, called HySI, operating in the 380–1000 nm range based on a tunable filter as monochromator, patented in 2006 [88]. The spectral range was extended to 1550 nm within the EU MUSIS project (Multispectral terahertz, infrared, visible imaging and spectroscopy, 2008–2012, FP7-ICT STREP) [89]. Liang et al. [90, 91] developed a MSI/HSI PRISMS (Portable Remote Imaging System for Multispectral Scanning) system specifically designed for remote imaging of wall paintings. The device operates in a VIS/NIR (400–880 nm, 10 filters, FWHM = 40 nm in the VIS, 70 nm in the NIR) as well as in a SWIR (900–1700 nm, best spectral resolution = 10 nm) range. Recently, Delaney et al. [38] combined two commercial HS cameras to cover the 441-1680 nm spectral range (260 bands). Lastly, the authors have extended the spectral range out to 2500 nm. [92, 93]

This scenario proves RIS for the investigation of works of art has reached the stage of mature technology, evidenced by the appearance of a few devices on the market [94]. Nowadays, MB, MS and HS devices, all suited for the non-invasive diagnostics and documentation of painted surfaces, differ in terms of transportability, price, spectral sensitivity, spatial and spectral resolution, ease-of-use, dimensions, weight, user-friendly interface, etc.). High spatial resolution is a crucial parameter for obtaining high quality images, whereas the possibility to identify pictorial materials strictly depends on the spectral resolution and on the extent of the spectral region investigated. At the same time, increasing the sampling step both in the spatial and in the spectral domain entails huge data sets as well as long acquisition time. Therefore, a compromise between high-quality data and ease of data management should be reached. [36]

The introduction of MSI/HSI in heritage science boosted an informatics research branch focused on pigment identification and mapping [38, 95,96,97], automated classification of data [98, 99], materials characterization [100], virtual restoration [101,102,103], machine learning analysis [104], automatic registration and mosaicking [105], web visualization and archiving [90, 91, 106, 107].

3 Instrumental set-ups for imaging spectroscopy

Imaging spectrometers generate digital images in a number of contiguous spectral bands. Therefore, the HSI/MSI data collection can be seen as a three-dimensional (3D) system, consisting of two spatial (XY) and one spectral ($\lambda$) dimension. By considering only a single spectral dimension, we exclude UV fluorescence and IR luminescence spectral imaging, which feature two spectral dimensions describing both the excitation and the emission wavelength. Hereinafter, a spectral dataset, with two spatial and one spectral dimension, will be indicated with the term spectral imaging cube or simply spectral cube or data cube.

To collect a stack of spectral images requires acquisition in both the spectral and the spatial domain. In the most schematic approach, a HSI/MSI system for the survey of artworks is generally composed of:

• 1. a lighting system (necessary for indoor applications),

• 2. optics (either lens or mirrors based, for the collection of the radiation),

• 3. a device for the spectral division of the radiation: a connotative element that characterizes the spectrometer and affects different instrumental designs,

• 4. a detector.

Many are the taxonomies for the HSI/MSI systems, basing on the different working principles and components. The sequential andsimultaneous spectral collection of images is respectively referred to as monochromatic orpolychromatic. One can further distinguish betweenhomogeneous and heterogeneous acquisition. In the homogeneous acquisition, the same spectral filtering is applied to all sensor elements in each recording. In the latter case, the light, reaching different sensor elements in a recording, is filtered in different ways. Simultaneous spectral collection can be performed in a heterogeneous modality if different filters are placed in front of individual pixels [108], entailing the monochrome CCD sensor modification, which can be expensive.

3.1 Wavelength filtering

Systems based on an image sensor acquire a near-instantaneous snapshot of an area without scanning. Using a lens to form an image at the focal plane, at which the image is sharply defined, and a set of filters for wavelength selection, are called staring, framing or wide-field systems, see Fig. 4

Fig. 4

Wide-field acquisition with $\lambda$ selection through filters

In this case, the 2D detector acquires simultaneously the two spatial dimensions of the data cube and the spectral information is acquired in a time interval [109, 110]. The main disadvantage of filter-based spectral imaging is the low light throughput and prolonged imaging times due to the fact that only one filter channel is collected at a time and the rest of the photons are discarded. Filter-based wavelength selection can be achieved through the light path of

• the illumination (before the light hits the object) or

• the imaging (between the object and the sensor).

In the first case, only a selected wavelength range, obtained by placing interference filters in front of a white source (quartz-tungsten halogen lamp), is incident on the object and the backscattered radiation is collected by means of a monochrome off-the-shelf digital camera. The main drawback of this configuration is the chromatic aberration entailing that images collected at different wavelengths have different focal lengths, resulting in a different magnification [111] that needs to be corrected in a post-processing. An option not requiring the use of filters is the use of Light Emitting Diodes (LED) generating sequences of spectral bands [112].

In the second case, wavelength selection on the imaging light path can be performed through filtering or dispersing elements, as well as Fourier Transform (FT) interferometers. Band-pass interference filters are generally interposed between the lens and the detector [34, 44, 51, 53, 68], mounted on a motorised computer-controlled filter-wheel. As a matter of fact, placing a filter in front of the lens is unpractical due to vignetting, unless filters of large numerical aperture are chosen, which usually are not from stock. There is a trade of between their low cost and slow speed and a limited number of wavelengths. Moreover, a small tilt in their position results in a conspicuous image shift. Tunable filters such as

• Acousto-Optical Tunable Filters (AOTF),

• Liquid Crystal Tunable Filters (LCTF), and

• Fabry-Perot etalon

are more expensive than interference filters, but allow for a much higher tuning speed and number of wavelengths [113, 114].

An AOTF consists of an anisotropic birefringent crystal bound to a piezoelectric transducer generating a high-frequency vibrational (acoustic) wave that propagates into the crystal, in response to the application of an oscillating radio frequency (RF) signal. Due to the photo-elastic properties of the crystal, the travelling ultrasonic (US) wave produces a periodic modulation of the refractive index generated by the alternating planes of compression and rarefaction: the crystal acts, then, as a transmission grating, causing the diffraction of light. Changing the frequency of the transducer signal applied to the crystal alters the period of the refractive index variation, and therefore, the wavelength of light that is diffracted. The result is an optical filter electronically, tunable over a wide spectral range (from the ultraviolet to the long wave infrared), which is compact, with no moving parts and fast frequency-tuning. However, the full spectral range is limited by the angular aperture (the larger the angular aperture, the smaller the full spectral range), and the possible crystal heating causes a shift in the diffracted wavelength [115].

A LCTF filter is essentially an electrically variable retardance, which provides the means to tune the center wavelength by changing the voltage applied onto the filter. It works on the principle of polarization dispersion, that is a broadening of the input pulse generated by a phase delay between input polarization states. It is based on a Lyot filter, which is a birefringent liquid crystal sandwiched between two polarizers whose axes are parallel to each other. The input polarizer converts incoming unpolarized light into linearly polarized light, which passes through the birefringent crystal that splits it into ordinary and extraordinary beams. The crystal also introduces a phase delay between the two beams (and hence, a retardance) proportional to the wavelength. By changing the voltage, selection of a specific wavelength, at the exclusion of all others, is achieved. LCTFs have relatively large acceptance angle and apertures, good compactness, simplicity of driving and no moving parts, but the spectral tuning range of this type of filter is narrower than those of the AOTFs [116].

The Fabry Perot etalon is composed of two plane, parallel highly reflecting and partially transmitting surfaces, with a given spacing between them. A beam incident at given angle generates a number of secondary reflected and transmitted beams of decreasing intensity. Light waves travelling a large number of round-trips between the two mirrors interfere constructively or destructively, depending on the phase relation. Spectral tuning is performed by adjusting either the reflecting surfaces spacing or the angle of incidence the input beam with respect to etalon. Limiting characteristics of the tuned etalon spectral imagers are: low throughput (consequent need of multiple frame averages to get a good signal to noise ratio - SNR), limited field of view (FOV) and sensitivity to temperature variation [117].

3.2 Wavelength dispersion

Wavelength dispersion can be achieved by means of a prism or a grating: the former is a refractive element working in transmission, whereas the latter is a diffractive element working in either reflection or transmission. Generally, designs based on prisms tend to be considerably more complex than the grating-based ones. Dispersive spectrometers can offer a higher spectral resolution with respect to filter-based systems, at the cost of a high demand in the chromatic aberration tolerance of the lens. By coupling the dispersing grating or prism with a 2D array of detectors, a spectral image is formed such that the spatial information is along one axis and spectral information is along the other. The dispersing element is combined with a slit through an input optics to focus the radiation on it, a collimating optic to bring parallel rays to the dispersing element, and then a focusing optic to focus the dispersed radiation on the detector. The slit is an opening that controls how much light enters the spectrometer. It is critical to the spectrometer’s performance: the amount of light (photon flux) that enters the optical path and the spectral resolution, i.e. the narrower the slit, the higher the resolution and the decreased the signal strength. These two factors must be balanced when selecting a slit size, and careful matching of the detector size and the slit is necessary for optimum performance [118]. The resolving power of a prism spectrometer is proportional to the base-length of the prism and to the variation in the refractive index with wavelength: the larger the prism and the higher the dispersion of the material, the finer the spectral resolving power. However, highly dispersive materials have high absorption, producing a reduction of the throughput of the system. In a grating spectrometer, the spectral resolution is proportional to the order and to the number of lines in the grating and is constant on the image plane for a constant incident angle of the radiation. Generally, designs based on prisms tend to be considerably more complex than the grating-based ones. Dispersive spectrometers can offer a higher spectral resolution with respect to filter-based systems, at the cost of a high demand in the chromatic aberration tolerance of the lens.

FT spectrometers are based on either a Michelson or a Sagnac interferometer: the common principle consists in splitting the radiation from a source into two beams and recombining them, after introducing a controlled phase shift. The combined beam is focused on a detector and the intensity of the light is modulated by the path difference of the two beams. In the former interferometer, the two mirrors are orthogonal to the optical path of the respective wavetrains and measurements at multiple optical path difference (OPD) values are obtained by scanning one mirror. In the latter interferometer, the two mirrors are not orthogonal to the optical path of the respective wavetrains, but they have a fixed angle between them, a fixed position (no moving parts) and identical distances from the beamsplitter. The reflected and transmitted rays follow exactly the same path, but in opposite directions. The beamsplitter provides the phase shift between the two coherent interfering rays so that the optical path difference changes linearly with variation of the angle of the entering ray with respect to the instrument optical axis [91, 114, 119]. The Sagnac spectrometer has relatively low resolution, but good mechanical stability and compactness (the lack of moving parts significantly reduces the sensitivity to vibrations). In a Michelson spectrometer, the spectrum is built up in a time interval: the finer the spectral resolution, the longer it takes to collect the interferogram; the lower the temporal resolution, the higher the SNR. In both cases, a FT to recover the spectrum is needed [120].

Whatever wavelength selection is achieved, scanning-based imaging spectrometers, which employ a sensor with a narrow FOV that sweeps over the area to build up and produce a 2D image of the surface, fall into two basic categories: whiskbroom and pushbroom scanners.

3.3 Whisk- and Push-broom Scanners

Whiskbroom devices scan the surface in a series of lines oriented perpendicular to the direction of motion of the platform as seen in Fig. 5, which coincides with the flight direction for airborne/spaceborne operation, i.e. across the swath. These systems are then also referred to as across-track, spotlight or single-point scanners. Each line is scanned from one side of the sensor to the other, using a rotating mirror to collect measurements from one pixel in the image at a time, and as the platform moves forward over the surface (an active scan mechanism is required to scan the spatial dimension orthogonal to the direction of the motion of the platform), successive scans build up a 2D image of the surface.

Fig. 5

Whiskbroom (single-point) scanning mode

Therefore, the surface is scanned point by point and line after line, following a boustrophedon movement. The instantaneous FOV of the instrument determines pixel size. The light from each pixel is passed through a dispersion element and measured using a linear (1D) array, aligned such that radiation of different wavelength ranges falls on different elements of the array. For each platform position, radiation is dispersed and its intensity is recorded in as many spectral channels as the number of detector elements in the line array [121]. The moving parts render this type of sensor expensive and more likely to wear out. A special device that combines the whiskbroom scanning with filtering is the multispectral scanner developed at the National Research Council–National Institute of Optics (CNR-INO). Simultaneous wavelength collection is performed through a square-shaped fibre bundle, which collects the reflected light from a single point and distributes it to a set of photodiodes (Fig. 6), each equipped with a different interferential filter. Basing on illuminating a very small area around the measured point, light scattering and surface heating are reduced. Moreover, the use of catoptric lens and single-point detection ensures aberration-free images, compensating for the limited speed of capture [58,59,60,61, 66].

Fig. 6

Single-point acquisition through a set of 32 photodiodes (multispectral scanner CNR-INO). The red circle highlights the 5 cm diameter area around the measured point (250 micron)

Pushbroom scanners also use the forward motion of the platform to record successive scan lines and build up a 2D image, perpendicular to the platform direction. However, instead of a scanning mirror, they use a 2D-array of detectors located at the focal plane of the image formed by the optics, which are “pushed” along in the track direction (i.e. along track): pixels are then recorded line by line, using forward motion of sensor (the motion of the detector array is analogous to a broom being pushed along a floor). Figure 7 shows as a separate linear array measures each spectral band or channel: the second dimension of the detector collects simultaneously all the spectral information [121, 122].

Fig. 7

Pushbroom (line) scanning mode

Compared to whiskbroom scanners,pushbroom devices are smaller and lighter and they have better spatial and radiometric resolution. Special provisions have to be taken to avoid or compensate any chromatic aberration of the imaging system over the entire wavelength range, in order to avoid a reduction of the spatial and spectral resolution of the system. For all devices, the usual trade-off of spatial resolutionvs. SNR and geometric distortions is valid. [123] With respect to wide-field imaging, scanning approaches provide improved spectral resolution by collecting a highly resolved spectrum at each point (whisk-) or line (push-) of pixels on the sample (Fig. 8). The spectral range of photographic systems is restricted to the VIS and NIR regions while MSI/HSI systems can extend this range into the IR. Photographic systems use separate optical systems (lens, filters) to acquire each spectral band, possibly causing problems both in ensuring that the different bands are spatially and radiometrically comparable, and in registration of the multiple images (post-processing to correct for distortion and aberration is necessary). MSI/HSI systems acquire all spectral bands simultaneously through the same optical system to alleviate these problems (even though not solving them).

Fig. 8

Acquisition modality: (left) wide-field and (center, right) whisk- and push-broom scanning

3.4 Detectors

The detector material sets the spectral sensitivity of the instrument as shows Fig. 9, together with the wavelength selection component. Silicon detectors are used in the 200–1000 nm range, and for low-noise detection, cooling systems are added (common cooling techniques involve thermoelectric i.e. Peltier, whereas cryogenic cooling using liquid nitrogen are used for very low-noise devices). For cultural heritage applications, the use of Indium Gallium Arsenide (InGaAs) detectors is widespread for their high sensitivity (900–1700 nm range) and low cost. Extended InGaAs detectors have sensitivity up to 2.5 $\mu$m but they require two-stage cooling for good SNR. Germanium (Ge) detectors have sensitivity similar to InGaAs detectors, but their use is hampered by the high cost. HgCdTe (also called MCT, mercury cadmium telluride) detectors are sensitive in a broad spectral range (1–10 $\mu$m, 2–20 $\mu$m depending on the element composition), but they are very expensive. For IR spectroscopy in the IR, other materials well suited are PbS (lead sulphide, 1–2.8 $\mu$m), PbSe (lead selenide, 1–4.5 $\mu$m) and InSb (indium antimonide, 2–5.5 $\mu$m), PbO-PbS (900 nm–2.2 $\mu$m) but are not used as frequently, mainly due to their high cost. Each arrangement has advantages and disadvantages, and it is the application/experimental requirement that defines which configuration is the most suitable (e.g. basing on gain, noise factor, response in a specific wavelength range of interest, ease of operation, necessity of liquid nitrogen cooling, light levels, etc.).

Fig. 9

Detector materials in relation with their spectral sensitivity

Optical detectors used in spectroscopic instruments can be either single- or multi-channel detectors (resp. SCD and MCD). SCDs have one active sensing element that acts as a single transducer. Photons reaching the detector, within its operating wavelength range, are absorbed by the active material and encoded as an electrical signal that varies according to the detector specifications (i.e. analogue or digital domain). In contrast, MCDs have multiple active sensing areas, which collect photons independently and simultaneously. A variety of array detectors are suited for image spectroscopy devices, such as photomultiplier tubes (PMT), avalanche photodiodes (APD), charge-coupled devices (CCD), intensified CCDs (ICCD), electron-multiplying CCDs (EMCCD) and scientific complementary metal–oxide semiconductor (sCMOS). Among them, the PMT is the most sensitive detector but suffers from a low dynamic range and high cost, prohibiting the manufacturing of high-pixel arrays (i.e. low-pixel (1$\times$32 and 8$\times$8 arrays are available at reasonable costs). APDs, though slightly less sensitive than PMTs, are much less expensive and exhibit superior dynamic range. CCDs are often used for spectral imaging: the large number of pixels in CCD arrays can be utilized for flexible tuning of spectral resolution by binning adjacent pixels (in the spectral direction).

High performance array detectors for the UV and visible region (CCD and CMOS) are readily commercially available with low background, high sensitivity at relatively low cost because they are based on well developed technology. Conversely, IR array detectors or IR cameras (also known as IR focal plane arrays, FPAs) were originally developed for military applications and surveillance and have only become commercially available recently. Furthermore, they are based on a technology that is relatively less developed than that for UV and visible cameras. However, as demand for these FPAs increases, concomitantly with recent advances in material sciences, optics and electronics, newer and high performance FPAs are being developed at much lower cost and larger size, better performance, wider spectral response, and faster read out.

4 Measured quantity

4.1 Reflectance

The reflectance is a parameter used for quantifying the ability of bodies, materials, or surfaces to reflect the radiation reaching them. According to the standard EN ISO 9488 [124], reflectance ($\rho$) is defined as the ratio of the radiant flux reflected from a surface, $\phi _r$, to that of the incident radiant flux, $\phi _i$,

$$\begin{aligned} \rho (\lambda )= \phi _r(\lambda )/\phi _i(\lambda ), \end{aligned}$$

(1)

where all quantities are expressed as a function of the wavelength ($\lambda$). Reflectance is dependent on the angle of incident and light polarization, as well, but considering the case of spectrometers with a fixed configuration geometry and not-polarized sources, the former dependence can be omitted. Being the ratio of two homogeneous radiometric parameters, $\rho$ is a dimensionless quantity ranging from 0 to 1. While the amount of reflected radiation is a property of materials, its angular distribution is related to the microscopic flatness or roughness surface of the material. Therefore, reflectance is distinguished into specular or diffuse reflectance. Specular reflectance, defined as the ratio of the energy flux reflected by a surface in the specular direction (i.e. at the same angle as the incident one) to the radiation incident on it, describes the case of reflection on a perfectly flat or planar surface (i.e. with roughness much less than the radiation wavelength) [125]. In this case, diffusion and diffraction phenomena are negligible. Diffuse reflectance occurs whenever the bundle of light is diffusely scattered in all directions in the plane of incidence. $\rho (\lambda )$ is defined as a function of the radiant flux, therefore the totality of the radiation reflected in a solid angle, $\Omega =2\pi$, has to be considered, which corresponds to a hemisphere limited by the reflecting surface.

To define the spectral reflectance factor - $R(\lambda )$, i.e. the output of a spectral reflectance spectroscopic measurement - the radiometric reflectance $\rho$ has to be referred to a perfect (ideal Lambertian) reflecting diffuser. Being this latter an ideal representation, a reference standard is used, whose reflectance is certified and traceable ($\rho _{ref}$). R($\lambda$) is then a relative parameter defined as the ratio between the spectral radiant flux reflected by the sample point, $\phi _{sample,r}(\lambda )$, in a certain direction and within a solid angle $\Omega$ and the spectral radiant flux reflected by a Lambertian reference surface, ($\phi _{ref,r}(\lambda )$), in the same illumination/observation conditions:

$$\begin{aligned} R(\lambda )= \frac{\rho _{sample}(\lambda )}{\rho _{ref}(\lambda )}=\frac{\phi _{sample,r}(\lambda )}{\phi _{ref,r}(\lambda )} \end{aligned}$$

(2)

For $\Omega \rightarrow 2\pi$, R($\lambda ) \rightarrow \rho (\lambda )$, the spectral reflectance factor of the perfect reflective diffuser is equal to 1.

In conventional image sensor, all pixel values are output once at the fixed interval (T) that is the maximum integration time. The output voltage, when measuring the sample surface (V $_{sample}$), is then proportional to T, following the formula

$$\begin{aligned} V_{sample}(\lambda )= k\phi _{sample,r}(\lambda )T+V_{dark}(\lambda ), \end{aligned}$$

(3)

where k is a constant value that depends on the detector and $V_{dark}$ is measured in the same lighting and geometric conditions as the sample, but closing the entrance of the instrument with a black stopper/cap. The dark image represents the dark current of the detector, therefore it is considered as a fixed bias that has to be eliminated when acquiring images. The same relation holds when measuring the reference voltage, i.e.

$$\begin{aligned} V_{ref}(\lambda )= k\phi _{ref,r}(\lambda )T+V_{dark}(\lambda ). \end{aligned}$$

(4)

During the reference standard acquisition, it is important not to saturate the detector in order not to lose linearity. In case of saturation, lower exposure time or reduced aperture should be used. The spectral radiant fluxes for the sample and reference standard (resp. $\phi _{sample,r}$ and $\phi _{ref,r}$ derived from the equations 3 and 4), substituted in the R($\lambda$) definition (Eq. 2), result in the following relation:

$$\begin{aligned} R(\lambda )= \rho _{ref}(\lambda ) \frac{I_{sample}(\lambda )-I_{dark}(\lambda )}{I_{ref}(\lambda )-I_{dark}(\lambda )}, \end{aligned}$$

(5)

where the true spectral reflectance of the spectral standard $\rho _{ref}(\lambda$) has been introduced as its value is strictly equal to 1 at every $\lambda$ just for the ideal perfect reflective diffuser. The parameterI (light counts) is used instead of V, having a direct proportionality. The quantity in the denominator is the average computed over a set number of pixel. To obtain optimal results, the instrument should be in the same setup (lens, device output bit depth, gain, ROI, down-sampling, light conditions, etc.) during the acquisition of calibration images (I $_{sample}$, I $_{dark}$, I $_{ref}$).

The measurement has to be performed in a specific illumination/detection geometry [126], which for the majority of instruments for reflectance imaging spectroscopy is $45^{\circ }$/$0^{\circ }$: light impinges on the sample at $45^{\circ }$ and the back-scattered radiation is collected at $0^{\circ }$ (all angles are measured with respect to the axis orthogonal to the surface). For object colour measurement, deviations from the standard geometry can cause significant error. To ensure such errors are not significant, the tolerances for the standard geometries are recommended in the CIE International Standard [127].

The acquisition of the spectral image of a scene (the spectral cube) requires the measure of the spectral reflectance factor in every pixel. Depending on the instrument, a few correction factors have to be introduced: the flat-field correction (FFC) is used to correct the differences of light sensitivity between the pixels of a camera, the differences in the transmission of light through the lens and the differences of illumination intensities in the field-of-view. FFC corresponds to correcting the combined optical-system and CCD throughput at each pixel so that each pixel on the CCD would respond equally to a source with the same photon flux. Flat fielding removes the effect of the pixel-to-pixel sensitivity variations across the array as well as the effect of dust or scratches on the CCD window, and vignetting. A flat field frame must be produced for each filter/dispersion element—lens—detector combination. The goal is to correct the pixels of the captured (raw) images in such a way that when a uniform background is captured by the system (camera and lens), the resulting output image is uniform. The flat image is an image of a uniform target covering the whole FOV: once the lens aperture and the illumination intensity are adjusted to obtain the brightest possible image providing that no pixels are saturated not to enter the non-linear response regime, several images are acquired and the average response for each pixel of the array is computed (the more and brighter frames, the better). [91] Detector nonlinearity is another source of error: if the nonlinearity is found as a function of integration time or count level for the pixels, the raw signal (counts) can be corrected using a fit function. If no correction is made, they should be included in the uncertainty budget. Another error source occurs when a grating has a leaked higher-order response: it may be useful to check the non-existence of the second-order diffraction. Stray light can cause significant error in chromaticity: this is critical for colour sources that have no emission in some spectral regions, such as LEDs.

Bandpass of spectrometers can cause significant errors in colour measurement if the bandwidth is larger than 5 nm. Even with a 5 nm bandwidth, the errors can be considerable for special samples and for applications that require low uncertainties. Some methods of correction for bandpass error are introduced. For colorimetric computations, in order not to apply any data deconvolution, CIE suggests a spectral bandwidth equal to the spectral sampling, both $\le$ 5 nm. This is usually the case for HS devices, whereas for MS data the Stearns and Stearns rule has to be applied [128], as well as 1 nm interpolation. Other methods are also available, depending on the spectral data interval and bandpass shape of the instrument [129].

MS instruments based on filtering suffer from a scaling error among images, in case filters do not have the same optical thickness. Moreover, a tilt in the position of the filter by a fraction of a degree can result in an image shift. Shift between wavelength channels is not present for tunable filtering. Chromatic aberration affects both MS and HS systems: standard lenses do not focus all wavelengths on to an image plane at the same location, especially in the IR range where also low dispersion glasses hardly compensate on a huge spectral range. The multispectral scanner developed at CNR-INO [66] overcomes all the above-mentioned corrections: based on the point-wise detection, the device images a sample point on the measured surface through a catoptric lens, avoiding the chromatic aberration. The ROI is very small (a square 1.5 mm wide) compared to the lighted spot (about 5 cm diameter), therefore uniform lighting is guaranteed. The image is formed onto the entrance end of a fiber bundle (6 by 6 fibers in a square-shape array as shown in Fig. 6) whose output end is filtered by a set of 36 interferential filters placed in front of 36 separate detectors. Therefore, no FFC is necessary. The lighting systems is integral with the collecting optics, performing a boustrophedon movement across the surface, minimizing thus surface heating (maximum temperature increase is 2.5 $^{\circ }\hbox {C}$ during scanning). Due to the poor spectral bandwidth and sampling, the Stearns and Stearns formula as well as 1 nm interpolation are necessary for colorimetric computation.

4.2 Reflectance spectroscopy

Reflectance spectroscopy (RS) measures the spectral composition of the radiation specularly or diffusely reflected by the object surface. RS usually deals with diffuse radiation therefore the experiments are planned so as to avoid the specular component. RIS in the visible to near infrared (400–1000 nm) provides information on electronic transitions. Observed electronic transitions (i.e., between different electronic levels) can be classified as follows [40]:

• 1. Between delocalized molecular orbitals. These transitions are characteristic of organic dyes and pigments having conjugate double bonds, possibly with electron-donor/electron-acceptor pairs far from each other, permitting electron delocalization over a wide space interval. Their energy levels are separated by energies corresponding to those in the VIS range thus generating very intense transition bands. Some examples include indigo, alizarin, curcumin.

• 2. Charge transfer transitions, in which a real electron transfer occurs between molecular orbitals localized in different sites of the molecule or crystal.

• 3. Ligand field transition, taking place between levels localized mainly on (transition) metal ion constituting elements of the material such as inorganic pigments (i.e., azurite, malachite, verdigris, smalt, cobalt violet, Egyptian blue) or gems (i.e., ruby, emerald). Five d orbitals of the metal ion are acted upon by the field generated by the ligand ions that surround the metal ion. A typical example can be Co$^{2+}$ ion coordinated by 4 oxygen atoms in a near tetrahedral geometry in smalt, generating transition in the VIS region.

• 4. Energy band transitions occurring in metals and semiconductors. The molecular orbitals resulting from the interaction of the atomic orbitals of each atom are close one to another to yield a continuum, i.e. energy band. In semiconductors, the valence (filled) and conduction (empty) bands are separated by the energy gap. The energy quantum corresponding to such energy gap is than the starting point for a continuous absorption of energy to yield a “S” shaped absorption band.

Extended range beyond visible to near- and short-wave (SWIR) infrared spectrum up to 2500 nm enables access to vibrational overtones and combinations, which are distinctive of functional groups such as carbonates, hydroxyls, sulphates, and methyl groups, facilitating the identification of number of compounds. RIS is thus providing information about the chemical structure and presence of certain functional groups for both organic and inorganic compounds. The work by Rosi et al. [130] demonstrated the utility to exploit even longer wavelength (mid-infrared) HS imaging.

The absorption bands seen in the NIR spectral range arise from overtones and combination bands of stretching and bending vibrations (Fig. 10). The combinations bands are the sum of several different fundamental vibrations whereas the overtones are multiples of fundamental vibrations frequencies. Their intensities are weaker by a factor of 10–100 and are inversely proportional to the order of overtone.

Fig. 10

Scheme of overtone and combination band spectral features. Modified from [133]

Reflectance maxima can be considered for a first inspection, although more reliable information derives from apparent absorbance maxima, which can be obtained through Kubelka–Munk or Log(1/R) transformation. Every spectrum is described as a two columns set of data, containing wavelengths and reflectance values, respectively. Pigment identification using visible reflectance may be hampered by the high dependence of RS on (i) the pigment particles size, (ii) the pigment/binding ratio, (iii) the type of binding medium, and (iv) scattering and autoabsorption phenomena. These factors can contribute to a change in spectral shape, a shifting of the bands, and a variation of relative band intensities. The presence of varnish and dirt can affect the reflection measurements due to absorption of reflected light [131]. Unsupervised methods, such as Hierarchical Cluster Analysis (HCA) or Principal Components Analysis (PCA), can be used to classify reflectance spectra into different classes [132]. Classification should be performed among painted areas of a same hue, otherwise differences among hues would prevail over differences among colourants of similar hue. Data transformation can be crucial to obtain significant results, for example a simple Z-scores transformation can be useful to avoid differences induced from the surface roughness of the sample. Normalisation should be performed along spectrum rather than along wavelength, in order not to lose typical spectral features. Another useful method is to perform classification on Log(1/R) data instead than on R data: this accounts for differences induced by colourant concentration.

RIS is successfully applied to study a variety of materials, including painted surfaces, paper, parchment, textile and plastics and their degradation products and mechanisms. It is especially suitable for identifying blue, green, white and red pigments as most yellow and black pigments do not exhibit sufficiently distinct spectral features (except for iron-based pigments). Binding media and varnishes can be generally classified, however, attention must be paid to account for the contribution from the support. Textile fibres—i.e. wool, silk, cotton and bast, constituting canvas, tapestries, historic carpets—can be distinguished as well. Paper can be discerned from parchment, and degradation mechanisms of the two materials can be followed. RIS can also contribute to the characterization of materials composing contemporary art. [134]

Hyperspectral data volumes are very large, implying certain requirements for storage and further image processing and analyses. The primary aim of data analyses is therefore a reduction step to decrease data size. To process the huge amount of data, multivariate analysis methods such as Principal Component analysis (PCA), Maximum Noise Fraction (MNF), Non-negative Matrix Factorization (NMF), Spectral Angle and Spectral Correlation Mappers (SAM and SCM), automated classification methods, such as neural networks, t-Distributed Stochastic Neighbor Embedding (t-DSNE) and Unifrom Manifold Approximation and Projection (UMAP) for dimension reduction (UMAP) are applied to group and map artist’s materials based on their spectral features [135].

The pioneering work by Baronti et al. [43] applied PCA to map pigments of a 16$^{th}$ century Italian painting, however the association of principal components with specific pigments or physical quantities is not straightforward. Comelli et al. [137] integrated the information gained using PCA with that obtained through SAM algorithm to study in depth Italian Renaissance paintings by Masolino da Panicale. The method further developed into the application of the automatic end-member extraction obtained with sequential maximum-angle convex cone (SMACC) to be complemented with the manual selection of additional end-members to completely describe complex spectral data sets. In fact, Cesaratto et al. [131] demonstrated that MS imaging data with SMACC and SAM algorithms are efficient in segmenting data into different areas that reflect diverse composition.

SAM and SCM algorithms are indeed among the most commonly applied methods providing information about the distribution of the pigment across the painting, as documented in Sect. 5 (RIS in multimodal analysis). SAM considers spectra as vectors in N-dimensional space, where N corresponds to the number of spectral bands. The angle between the reference (r) and target (t) spectra, Eq. 6, measures their similarity. The angle is used to build the similarity maps, where a smaller value, expressed in radians $<0,\pi>$, suggests a higher degree of similarity. [131, 136]

$$\begin{aligned} \alpha =cos^{-1}\frac{\Sigma _{\lambda }r_\lambda t_\lambda }{(\Sigma _{\lambda }t^{2}_\lambda )^{1/2}(\Sigma _{\lambda }r^{2}_\lambda )^{1/2}} \end{aligned}$$

(6)

The main difference between SAM and SCM is that the latter centralizes the data in its mean (Eq. 7) considering also the negative correlation. In fact, SCM relies on the calculation of similarity ($-1<R<1$), where 1 means total correlation, through the Pearson’s correlation coefficient, yielding more accurate classification results.

$$\begin{aligned} R= \frac{\Sigma _{\lambda }(r_\lambda -\overline{r})(t_\lambda -\overline{t})}{(\Sigma _{\lambda }(r_\lambda -\overline{r})^2)^{1/2}(\Sigma _{\lambda }(t_\lambda -\overline{t})^2)^{1/2}} \end{aligned}$$

(7)

Spectral Angle Mapper (SAM) considers spectra as vectors with dimensionality corresponding to the number of bands. By calculating the angle between the reference and target spectra, it measures their similarity. In SAM images, the pixel value intensity is proportional to the angle between the vector representing the spectrum of each pixel and the endmember (EM) being mapped. A small angle means a close match and a high intensity value in the image plot. Therefore, the SAM algorithm identifies pixels in the image cube whose reflectance spectra match those of the reference (or EM) within a specified angle tolerance.

Many researchers rely on ENVI software (ENVI, Exelis VIS, Boulder, CO) with implemented algorithms. [138,139,140] One of the most used is the hourglass paradigm that entails three major steps. The first consists in the determination of the spectral diversity through the minimum noise fraction (MNF) algorithm. This returns the eigen images, orthogonal projections of the pixels in the image cube, whose maximum numbers corresponds to the number of spectral bands. Through the purity pixel index algorithm, in the second step, the image pixels are grouped based on their spectral similarity. A convex geometry hyperspectral processing algorithm can be used to find a spectral basis set (i.e., EM) of pigments and binders. The method utilizes both PCA to determine the diversity of the image cubes and convex geometry to find unique spectral components. The reflectance spectra identified by the convex geometry algorithm are manually clustered in principal component space [141]. The resulting hyperspectral maps show the spatial distribution of the EM. The maps are false-colour images where each colour denotes one endmember, and the colour intensity represents the degree of match. The histogram of the image maps are set to values of 0.1–1.0 (i.e. poor to high degree of match).

5 RIS in multimodal analyses

In the last years, RIS has been used in combination with other non-invasive/non-destructive techniques to achieve a comprehensive knowledge of the analysed artwork. As pointed out in previous reviews [91, 142, 143], this multi-analytical approach has often involved the application of punctual, areal and 3D analyses, to provide chemical, morphological, and structural information on the object and its constituting materials. Most studies aimed at obtaining detailed information for the chemical characterization and mapping of painting materials- i.e. pigments, binders, colorants and smalts-, and this was attained by combining punctual spectroscopic analyses with complementary point-wise and areal measurements. 2D and 3D imaging techniques were often applied jointly to visualize hidden details underneath the paint layers, identify and localize retouched areas, and determine the state of conservation of the object through the detection of defects inside the material structure. Finally, 3D methods provided shape models and cross-sectional images of painted artworks, turning useful for deeply understanding the artistic technique and the object’s genesis, as well as monitoring its structural and morphological condition.

In this section, we report on the state of the art synergic application of RIS in the UV–VIS–NIR-midIR region and other non-destructive techniques for the analysis of artworks. The overview covers the main focus and objectives reached by each research line in the last decades subdived as follows:

• Point-wise and areal analyses of painting materials

• 2D imaging of hidden details

• 3D and cross-sectional survey.

Firstly, contents of each section are briefly summarized, and then case studies are reported in chronological order.

5.1 Point and areal analyses of painting materials

The non-destructive identification and mapping of the constituting materials is often based on MS/HS reflectance point-wise and areal analyses. These are often combined with other point-mapping and/or wide-field imaging techniques, such as Raman spectroscopy, X-ray fluorescence (XRF) spectroscopy, UV–VIS spectrophotometry, and Fourier-Transformed Infrared (FT-IR) spectroscopy. Data post-processed by means of multivariate analysis can reveal the areal distribution of the materials. In some cases, micro-destructive technique—i.e. involving micro-sampling, such as scanning electron microscopy/energy dispersive spectroscopy (SEM/EDS) and polarized light microscopy (PLM)—are used to confirm compositional information provided by RIS.

The combined application of complementary spectral point-wise techniques, namely UV–VIS reflectance spectroscopy, Raman and FT-IR microspectroscopy, demonstrated that such multimodal approach may provide a rapid and unequivocal way to identify ancient pigments [76]. The pre-Columbian pigment Maya blue was in fact identified on archaeological samples, based on the detection of significant spectral differences between indigo as pure crystalline solid or as complexed by palygorskite. Later, a selection of 12$^{th}$ century manuscripts was analysed by means of IRR, Raman spectroscopy (in situ and laboratory measurements), XRF, and UV-fluorescence photography in order to localize the presence of retouched areas on the basis of the chemical composition of the pigments. Apart from historical pigments, two synthetic azo-pigments, PR4 and PR176, could be identified, suggesting that the manuscript underwent some restoration in the past, even though no previous conservative intervention was documented [144].

In 2010, primary colorants and, in some case, pigments mixtures used by Picasso in the painting Harlequin Musician were identified by combining VIS-SWIR HS and MS luminescence imaging [38]. Reflectance spectra of colorants displaying unique luminescence properties, called endmembers, were used to compute the associated spatial maps by means of HS image processing algorithms (Fig. 11). Results were compared with those from other more traditional methods, such as in situ XRF and polarized light microscopy (PLM) on micro-sample analysis, SEM/EDS, and FT-IR. It was observed that many of the primary colorants could be effectively identified and mapped by RIS only when extending the spectral range into the SWIR and when including information obtained with luminescence IS. In fact, the omission of part of the NIR and SWIR portion of the reflectance image cube was found to reduce the degree of discrimination among the colourants. Furthermore, the assignment of pigments was confirmed by complementing the results of reflection and luminescence IS with XRF data, enabling to characterize and map the pigments used by Picasso.

Fig. 11

P. Picassos Harlequin Musician (1924): (left) VIS colour image; (top left) VNIR/SWIR HS cube and (bottom left) cluster map used to define the primary end members; (centre) A plot of the reflectance spectra of the end members and (right) their associated spatial distributions in the painting. Modified from [38]

In 2011, Thoury et al. [145] attained the non-invasive identification and mapping of cadmium pigments in painting, based on the combined application of FORS and luminescence IS. A homemade illumination and MS camera system operating from 650 to 950 nm provided image cubes calibrated to the relative radiance of test panels and paintings. The unique spectral properties of pigments highlighted by this method enabled to map the distribution of cadmium pigments of different hues in a painting by Steichen (Fig. 12a), which was confirmed by XRF. Luminescence spectra across the painting were extracted from images collected in seven spectral bands from 650 to 950 nm while exciting with blue-green light. The false colour image reported in Fig. 12b, constructed by using three bands at 650 nm (blue), 800 nm (green), and 950 nm (red), revealed a varied and complex set of emissions in the red to NIR range, showing the diversity of luminescent materials on the painting and giving insight on their relative luminescence yields. The spectral maps computed on the basis of the endmembers’ luminescence spectra well-described the spectral diversity of the various luminescent materials, thus providing a good discrimination among them (Fig. 12c).

Fig. 12

E. Steichen The Sunflower(image courtesy of the Board of Trustees, NG of Art, Washington, D.C): a visible image, black numbered points indicate XRF analyses; b false colour NIR luminescence image: 650 nm (blue), 800 nm (green), and 950 nm (red) formed using the registered and radiometrically calibrated spectral image cube using a linear contrast stretch; c spatial distribution of end members. Modified from [145]

In 2012, a protocol involving a sequence of complementary techniques was proposed for the non-invasive identification of materials in miniature paintings. [132] The combined application of UV-VIS reflectance spectrophotometry and optical microscopy followed by Raman spectroscopy and XRF enabled to characterize both the spectral properties of pigments and metal layers, verify the presence of overlapping layers, identify mordants in lakes, and recognize minor components yielding information concerning provenance.

The effectiveness of a synergic combination of point-like analytical and imaging methods for the study of complex polychromatic artworks was demonstrated by Cesaratto et al. [131] on a painting on paper by Van Gogh, analysed by MS imaging and micro-Raman spectroscopy. The corrected MS data cube of the entire painting was post-processed using multivariate statistical analysis, including the sequential maximum-angle convex cone (SMACC) algorithm to extract automatically the spectral endmembers. Then, the spectral angle mapper (SAM) was applied to visualize the similarity of the reflectance factors of the dataset with each endmember, thus enabling to differentiate the presence of pigment mixtures on the painting surface. Complementary analytical information was obtained through portable Raman spectroscopy, performed on a few selected points of the painting, corroborating the results obtained with reflectance spectroscopy and completing the characterization of Van Gogh’s palette.

The potential of whiskbroom line RIS in NIR range in providing insight into the complex mixing and layering of pigments was explored on a series of panel paintings by Cosimo Tura. [141] Mapping and assignment of organic (egg yolk and glue) binders was performed in situ. The proteinaceous component of both binders has similar spectral fingerprint in the region from 1650–2500 nm, whereas that associated with lipids differ. In fact, egg yolk can be discerned from glue through its lipid distinctive features localized at different wavelength and at higher intensity than for glue.

Each resulting image cube from RIS was post-processed with HS convex geometry algorithms providing the spectral endmembers. The analysis highlighted the presence of two binders, i.e. egg yolk and animal skin glue, whose distribution was ascribed to specific pigments on the basis of their X-ray density. Results, underscoring the strength of the imaging capability of RS, suggested that the artist used egg yolk as the binder for high X-ray density pigments (such as lead white and red/brown iron earth pigments), whereas glue was used for low X-ray density blue (mainly azurite) and white pigments (such as calcium carbonate).

In 2014, RIS in VIS-NIR range and XRF imaging was applied to identify pigments and determine their spatial distribution on another painting by Cosimo Tura [146]. The fusion of data from the two imaging modalities, based respectively on molecular and elemental spectroscopy, allowed for the disclosure of the preparatory design of the painting, as well as for the characterization of the materials with higher confidence than from either technique alone.

The analytical suitability of HSI, in combination with other non-invasive methods for identifying original pigments, was evaluated two years later on a number of paintings by Goya [147]. Two different HSI systems were compared in terms of their performance: pushbrooom providing high quality images, suitable for post processing, and mirror-scanning enabling the instrument’s mobility thanks to its low weight and dimensions (at the expense of low lighting uniformity). Data obtained with the two systems led to the identification of most pigments and the discrimination between retouched and original areas. Due to the presence of pigments mixtures, the characterization required the support of Raman Spectroscopy and XRF, with portable and hand-held devices, respectively.

Later, the same authors [148] explored the advantages of HSI over point-by-point spectroscopic analyses—i.e. handheld XRF and Raman spectroscopy. For that purpose, three paintings by Goya were examined to determine the best non-invasive technique for pigment identification. Each of the applied methods provided different and complementary information, thus demonstrating that a multi-modal non-invasive approach may effectively yield the comprehensive characterization of pigments on paintings. HSI gives general information facilitating the representativeness of the results regarding the entire artwork. In contrast, point-by-point analyses seem to provide more specific data.

Such approach proved successful for the study of illuminated manuscripts [149]. Data obtained with filter-based NIR RIS, XRF mapping, and Raman spectroscopy were post-processed with multivariate methods, including principal component analysis (PCA) and non-negative matrix factorization (NMF), to separate contributions from different pigments in complex images. On one hand, XRF mapping combined with Raman point analysis enabled to detect the spatial distribution of pigments in both surface and hidden layers. On the other, integration of XRF and RIS data allowed to identify all the locally investigated pigments.

Diffuse whiskbroom HS VIS-NIR RIS was applied jointly with macro XRF (MA-XRF) scanning and quantitative electron microprobe analysis to identify the presence of cobalt-containing materials on the painting Saul and David (Fig. 13a) by Rembrandt [150]. The strong electronic transitions of smalt in the NIR were used for mapping three types of cobalt-based pigments, assessed in comparison with XRF maps of Co, Ni and As. The distribution of smalt obtained by NIR RIS (Fig. 13b) on a painting’s detail overlapped only partially with the Co-K XRF map (Fig. 13c). In particular, the turban, from the left brow to the top right, showed strong Co XRF signal, but low smalt correlation in the RIS map. In fact, only the XRF map of the Ni-richer smalt found in Saul’s turban matched well with the smalt mapped by the RIS. Further observation with cross-sectional analysis revealed the presence of thick layers of smalt in the areas rich of Ni, while other pigments (such as iron ochres and black pigments) were present at significantly lower levels. It was concluded that, compared to the other areas where elemental Co was found by XRF, these high Ni content areas should have had an optical appearance closer in colour to that of blue smalt. Furthermore, the presence of two types of smalt not only supported the recent re-attribution of the painting to Rembrandt, but also revealed that the picture was painted in two phases.

Fig. 13

Rembrandts painting: detail of Sauls turban (a) visible light image (b) cross-correlation HS RIS map of smalt (threshold (0.5–1.0 confidence) normalized); c MA-XRF CoK map. Modified from [150]

In 2017, macroscale multimodal chemical imaging was exploited for the first time to shed light on the production technology of an 1800-year-old encaustic painting (Fig. 14a). [92] HS pushbroom line-scanning diffuse reflectance (400–2500 nm) and luminescence (400–1000 nm), and XRF (2–25 keV) mapping were processed in conjunction for non-invasive characterization of the heterogeneous painting system. This approach was found effective in disclosing the build-up sequences of the painting, the underdrawings and the original contours and forms of the depicted figures. Specifically, co-registred data cubes from the combined HSI modalities enabled the joint interpretation of reflectance, luminescence, and XRF features at each pixel in the image for the entire painting. Such merging of data allowed for a more thorough identification and mapping of the painting’s organic and inorganic constituents. Specifically, spectral endmembers extracted from HSI reflectance datasets gave a discriminating absorption feature indicative of wax with characteristic doublet at 2311 and 2352 nm, due to a combination band of asymmetric/symmetric stretching and bending of the methylenic (CH$_2$) groups (Fig. 14b, c). The abundance and widespread distribution of wax throughout the painting, except for areas of paint loss or areas having high concentration of black pigments, combined with the distinctive textured surface of the paint layer, indicated the use of melted beeswax as the binding medium mixed with pigments. Finally, the elemental MA-XRF mapping of Fe in specific areas revealed features in the painting not discernible by the naked eye or the HSI, due to the low reflectance value.

Fig. 14

Detail of Greco-Roman painting Portrait of a Woman, 2nd century AD a colour image, b reflectance spectral endmembers in the 2230–2360 nm spectral region,c chemical map of spectral signatures from wax-related endmembers. Modified from [92]

In the same year, NIR RIS in combination with MA-XRF found evidence for the presence of oil and alkyd paints in a “drip” painting Number 1 by Pollock (Fig. 15a). [139] Both the oil and alkyd spectra exhibit common features of a drying oil. The identification of the latter is based on presence of the aromatic C–H overtone at 1675 nm and aromatic C=C and C–H combination band at 2265 nm (Fig. 15b). Furthermore, integration of RIS with MA-XRF allowed associating the use of a binder to specific pigments (Fig. 15c). Reflectance properties highlighted in white paint spectra suggested the presence of a mixture of titanium white (anatase and rutile) and zinc white. Representative XRF maps from the white oil and white alkyd paints (Fig. 15c–f) supported this assumption, revealing the presence of both zinc and titanium, along with barium only in the oil paint. The observed high reflectance in the NIR of both zinc and titanium whites ensured that the NIR spectral absorptions used for mapping the oil medium were not influenced by the pigments present.

Fig. 15

Pollock’s Number 1: a colour image, b reflectance spectra of linseed oil in red and soy-based alkyd resin in blue, c false-colour image map of the endmember derivative spectra of oil and alkyd, in red and blue, respectively, corresponding to white paint; XRF element maps ofd titanium, e zinc, and f barium. Modified from [139]

A significant step towards the fusion of RS and XRF data for the analysis of painting materials was the development of an integrated imaging spectroscopic scanner capable of collecting both the RS and the XRF image cubes (Fig. 16a) [93]. In specific, a stationary, single pixel XRF and FORS spectrometers compose the scanner. The artwork is moved in a raster mode in front of the two detection systems through a 2D position-controlled easel enabling data acquisition at moderate spatial resolution (1 and 3 mm respectively for XRF and FORS). The set-up was characterized by a broad spectral sensitivity (400–2500 nm) in the FORS modality, overcoming the necessity of two HS reflectance cameras to cover such wide spectral range. The performance of the dual-mode device was demonstrated by analysing Haymakers at Montfermeil painting by Seurat (Fig. 16b). Despite the relatively low reflectance spatial sampling, satisfactory pigment identification and relevant distribution maps were yielded, yet some aspects as the positioning in focus could be improved (see false colour image in Fig. 16c). The complementarity of the two techniques enabled to identify cobalt blue, lead white and an iron oxide pigment in the analysed point reported in red in Fig. 16b), not possible by either of the single method. In fact, the strong NIR absorption of cobalt blue in the reflectance spectrum (Fig. 16d) nearly masks the weak absorption at 1445 nm, typical of the hydroxyl group of basic lead white. The latter pigment is instead perfectly detectable in the XRF spectrum (Fig. 16e), where the lead peaks are the most intense. Lower spatial resolution and longer collection times are obtained with such constructed dual imaging modality XRF and FORS scanner as compared with HS reflectance imaging cameras. Using FORS instrument makes, however, for a relatively lower cost.

Fig. 16

a Integrated FORS and XRF scanner. Seurat’s Haymakers at Montfermeil (1882): b colour image, spectra acquisition point marked with a red dot; c false-colour image (450, 550, 620 nm) created from FORS cube; single spectra from d FORS ande XRF data. Modified from [93]

Whiskbroom VIS-NIR MSI, complemented by XRF mapping, aimed at chemical and spatial characterization of pigments in two Madonna of the Rabbit paintings: by Titian (1525 and 1530) and by Manet (1856) [66]. In collaboration with the Czech Academy of Science, the researchers proposed the methodology based on advanced techniques of data-processing to analyze inhomogeneous datasets. This approach was demonstrated on datasets related to two aforementioned paintings examined in different contexts, times and partially different techniques. The spatially registered VIS-NIR data from the scanner, facilitated their processing by spectral correlation mapping (SCM) and feed-forward artificial neural network (ANN) algorithm (based on suppressing VIS information content from that contained in NIR), respectively for pigment mapping and improved visibility of pentimenti and of underdrawing style. The data provided several key elements for the comparison with a homonymous original work by Titian studied within the ARCHive LABoratory (ARCHLAB) transnational access project. By combining the results from XRF, spectrophotometric analyses and microscopic observation, it emerged that different blue pigments were employed by the two artists - i.e. lapis lazuli and azurite by Titian; Prussian, cobalt and cerulean blues by Manet. In Fig. 17 on left, SCM map of two main blue pigments in Manet’s artwork is shown. The FORS and Vis-NIR scanner spectra (Fig. 17, right) acquired on blue areas exhibit reflectance features ascribable to cobalt and Prussian blue, corresponding to the pale and electric blue areas, respectively, in the SCM map. The chemical composition was confirmed by XRF Co and Fe elemental maps.

Fig. 17

Manet’s Madonna of the Rabbit painting. (Left) SCM of cobalt (shawl) and Prussian blues (mantle) in pale and electric blue codification, (right) relative FORS and VIS-NIR scanner spectra. Modified from [66]

Fig. 18

Manet’s Banks of the Seine at Argenteuil painting (1874): a colour image; b false-colour showing the distribution of cobalt (red) and tin (blue);c false-colour image (R = 950 nm, G = 1230 nm, B = 1705 nm) showing cobalt blue in pink and cerulean blue in orange; d false-colour image showing the distribution of arsenic (red) and copper (blue). Modified from [151]

Results shown are in line with those reported by Amato et al. [151] on the paintings by Manet conserved in the Courtauld Gallery, investigated for the first time using a range of non-invasive in situ analyses. The non-invasive approach, including MA-XRF, RIS, UV–VIS–NIR, Raman and reflection FTIR spectroscopies, proved to be well suited to investigate the painting’s compositional complexity allowing for the visualisation of the distribution of elements indicative of the pigments used. It was found that Manet used cobalt and Prussian blues inDejeuner sur l’herbe, a painting almost coeval with Madonna of the Rabbit, whereas cerulean blue was detected in A Bar at the Folies-Bergere andBanks of the Seine at Argenteuil, along with the two aforementioned pigments. Results on the latter painting are reported in Fig. 18a–d. The distribution of cerulean blue was mapped by MA-XRF based on the co-localised XRF detection of cobalt and tin (Fig. 18b), while the pigment cobalt blue was conclusively assigned by reflection spectroscopy, thanks to its recognizable features. False colour images produced using NIR channels at 950, 1230, and 1705 nm enabled the discrimination of both pigments (Fig. 18c), based on their peculiar absorption properties. The use of green pigments such as Scheele’s green, emerald green, chrome green and viridian was suggested by the detection of copper, arsenic, and chrome XRF signals (Fig. 18d), which was confirmed by the absorption features highlighted by reflectance spectroscopy in the visible range. Finally, the combination of reflectance spectral data and antimony and iron XRF signals acquired on yellow areas enabled to identify Naples yellow.

A further multi-analytical approach involving complementary imaging and point-wise techniques examined the realization of two drawings on paper by Redon [152]. Both artworks date to the period in his career in which he was transitioning between two stylistic modes. RIS and MA-XRF complemented with Raman spectroscopy and FORS evidenced the benefits of a top-down research framework, whereby imaging techniques are employed first for the survey of the whole object, followed by investigation in areas of interest with site-specific methods, to non-invasively characterize the constituting materials. The identification and mapping of six black drawing materials demonstrated the potential of these complementary techniques to distinguish between similar components within a single object. Based on multivariate post-processing of spectral data, the array of drawing materials used by Redon was identified, many of which containing synthetic colorants.

In 2019, broad spectral range (BR) RIS— i.e. from the near UV through the mid–IR (350 nm-25 $\mu$m)—was tested as an in-situ imaging modality to provide maps of organic and inorganic artists’ materials in illuminated manuscripts. [140] Two separate, stationary spectrophotometers, each for a different spectral portion, composed the set-up: a FORS for the UV to NIR (350–2500 nm), and a FTIR spectrometer for the mid-IR (1818 nm-25 $\mu$m). The great amount of electronic and vibrational transitions within inorganic compounds and organic functional groups that can be detected in this spectral region makes BR-RIS a powerful tool for the identification and mapping of pigments, paint binders, fillers, and substrates used for lake pigments. To prove the potential of BR-RIS, 3D image cubes (2D spatial and 1D spectral) were collected first on mock-ups and then on early Italian Renaissance illuminated manuscripts (Fig. 19a) using this simple single-pixel scanner and a computer-controlled easel to move the artwork. Pigments and their associated binding media were identified by merging the spectral information obtained on the broad spectral range with macroscale imaging (Fig. 19) and the reliability of the results was confirmed with XRF spectroscopy. Of particular interest was the identification, through the (CH$_2$) combination band, of egg yolk tempera paint binder to realize the figure of prophet. The wavelength position of this hydrocarbon band can be used to discriminate binders as glue (2282 nm), drying oil (2304 nm), egg yolk (2309 nm), or wax (2312 nm).

Fig. 19

L. Monaco Praying Prophet:a colour image; b CaCO$_3$ map; overlay of the maps obtained the spectral features of c the pink lake and CaCO$_3$ (d). e The yellow lake and CaCO$_3$ (f); (g) the yellow lake, azurite, and CaCO$_3$ (h). Modified from [140]

Fig. 20

Roman fresco fragment (Pozzuoli): XRF elemental maps ofa K and c Fe, combined with the SCMs in b and d, points of spectra acquisition indicated in the RGB image e, yellow circle for K and black circle for Fe. From [153]

The study demonstrated that BR-RIS offers a larger amount of information for mapping artists’ materials than VIS-NIR, mid-IR, or XRF imaging spectroscopy alone. A clear drawback of the proposed system was the relatively slow scan rate (0.2–1 mm/s) for a 1–2 mm$^2$ spatial sampling (i.e. 1m$^2$ painting in several months). Suggested solutions are either decreasing the integration time for single-pixel scanners or multiplexing by increasing the number of spatial pixels collected at the same time from 1 pixel to 100 s. Scan rates of 10 mm/s for 1 mm$^2$ sampling would result in 30 h to scan a 1m$^2$ painting, comparable in time to high-quality XRF scanners. An alternative is the use of imaging spectrometers or HS cameras for portions of mid-IR, at higher costs due to increased complexity of the spectrometer and the mid-IR focal planes, that are currently, however, not providing the needed sensitivity. [140]

Dal Fovo et al. [153] applied three non-invasive transportable devices to analyse polychrome fragments of mural painting from two recently discovered Roman villas. MA-XRF elemental mapping, MS RIS, and FORS were exploited to display the distributions of the different pictorial materials, while assessing the chemical composition of the pigments present. MA-XRF elemental maps were complemented with the spectral correlation maps computed from the Vis-NIR images acquired with the multispectral scanner. The synergic use of the three techniques, as well as the correlation between point-wise data and post-processed images, enabled to acquire complementary spectral and chemical information, leading to the identification of different pigments. An example of this application is reported in Fig. 20. The main elemental components of the ivy crown on the man’s head analysed by MA-XRF resulted Fe and K, but with different distribution (Fig. 20a, c), bringing into question the use of the same pigment. The SCMs, based on the spectra acquired in the reference points highlighted in Fig. 20e, confirmed the presence of the same pigment in all the crown’s leaves, identified with green earth, based on its spectral features and the comparison with a reference spectrum. The results showed that SC mapping is an effective and fast imaging method that can be used to verify the distribution of a pigment on a wide pictorial surface or on several painted objects simultaneously. Based on the so-obtained spectral data, more detailed chemical analysis can be performed by MA-XRF on limited areas.

In a comprehensive survey on the painting Madonna of the Basket by Rubens and its presumed textile copy, a tapestry by Fevère, SCMs from MS-RIS, FORS and Raman spectroscopy shed light onto the artworks’ history [154]. Combined approach proved powerful in showing a good correspondence between the two artworks in terms of colour distribution of red pigments, while numerous differences were observed as regards the distribution and composition of green and blue colours. Furthermore, the geometric overlap of multi-spectral images provided evidence of the significant iconographic similarity, suggesting that Fevère may have indeed referred to the same carton used by Rubens for his oil painting. Finally, the ratio between the measured reflectance values in the 1500–1600 nm spectral range was used to discriminate between silk and wool used by Fevère to create shine and matte effects on the tapestry.

Recently, VIS-NIR MS data were fused with XRF single spot analysis for the examination of a painting by Carpaccio [155]. A workflow based on PCA analysis of integrated data aimed at mapping pigments in large paintings. After the fusion of XRF and RIS datasets, PCA was applied to evaluate the effectiveness of this combined approach compared to the PCA analysis of the two data-set separately. The first proposed method allowed to identify well-defined clusters linked to the different red pigments used by the artist and to the conservation history of the painting. These clusters were not detectable when analysing the dataset independently.

5.2 Two-dimensional imaging of hidden details

In the last years, the non-invasive investigation on multi-layered painted objects has involved RIS in combination with other imaging/mapping techniques for the visualization of underpaintings/underdrawings not visible to the naked eye. Generally, mosaicking algorithms and multi-variate analyses, while significantly improving the interpretation of the results, also demonstrated the complementary nature of various imaging techniques in revealing the presence of internal materials and hidden iconographic features. One of the first reviews on non-destructive optical techniques for the analysis of artworks, and specifically for the unveiling of underdrawings, pentimenti and hidden signatures, was presented in 2003 [156]. Main attention was paid to NIR reflectography to reveal underdrawings containing carbon, UV fluorescence to highlight retouched or newly repainted areas, and X-ray radiography (XRR) to give insight into older compositions. These methods applied jointly could provide exhaustive knowledge of details not visible to the naked eye.

In 2015, the effectiveness of combining different imaging modalities for the visualization of inner details in paintings led to the definition of a multimodal registration and mosaicking algorithm, enabling the accurate alignment of imaging data to reference colour images taken at high spatial sampling (300–500 pixels per inch). [105] By registering different data, such as MS IR images, X-radiographs, HS RIS and XRF image cubes, onto the reference images of the examined surface, several hidden features were revealed. The new algorithm was applied, for instance, to the X-radiograph and the IR image of a painting by Vermeer, which were superposed to the reference colour image. The IR image revealed an upside-down hat painted with an IR-absorbing pigment, whereas the X-radiograph showed an upside-down face of a man (in addition to the girl), partially painted with lead white. The accurate registration of these images allowed the construction of a composite image, which evidences both the man’s hat and face of the underlying composition. Thanks to the merging of IR image and the X-radiograph, it was observed that the painting style of the reversed portrait differed significantly from that of Vermeer, thus suggesting the authorship of a different artist.

The painting Belle Ferronnière by Leonardo was examined through a combined imaging approach, involving RIS and MA-XRF, to determine the state of conservation in view of restoring interventions and to deepen the knowledge of the artwork’s genesis [157]. Data provided by a new XRF scanning system complemented with RIS shed light on the technique to obtain the flesh tone, revealing the composition of the superimposed painting layers realized in two consequent steps. Furthermore, microscope examinations of specific areas unfolded materials’ alteration in the superficial layers, probably ascribable to a previous inappropriate cleaning. The integration of RIS and XRF approach proved powerful for a routine use in a museum environment to determine in this specific case the extension in-depth of such damage found to be confined to the upper layers. These results reinforced the decision to undertake a very subtle retouching of the damaged area in order to spatially reintegrate it.

A relevant disclosure of hidden paintings/underdrawings was presented by Thurrowgood et al. [158] on a painting by Degas. The analyses involved the combined application of conventional IRR, a high definition synchrotron radiation X-ray Fluorescence Microscopy (SR-XFM) and X-ray Raman scattering. The XRF elemental maps were post-processed with a novel methodology to achieve detailed technical understanding of the painting, which could not be resolved by conventional techniques. Specifically, basing on the traces visualized by IRR (Fig. 21a), the hidden portrait of a woman, attributed to the same author, was unveiled under the visible one. A false colour image of the underlying painting (Fig. 21c) was created using a methodology for layering multiple elemental maps. This latter was generated using custom-written software for merging the high resolution, high dynamic range elemental images (Fig. 21b), which were manually assigned with colours most likely associated with each element (e.g., red for Hg, blue for Co).

Fig. 21

E. Degas Portrait of a Woman:a IR reflectogram (detail); the obscured upside-down sitter’s face is indicated with a dotted line.b XRF elemntal maps providing an overview of the construction of the painting. c False colour reconstruction of Degas hidden portrait (detail), created from the SR-XRF elemental maps. Modified from [158]

The effectiveness of the synergic application of XRR and HSI in reflectance and transmission mode was further demonstrated by Van der Snickt et al. [159] by revealing underlying figurative composition in a painting by Magritte (Fig. 22a). A portable dual channel NIR (400–1000nm) and SWIR (1000–2450 nm) whiskbroom HSI in the transmission mode provided more information about the hidden pictorial features than imaging in reflectance mode. Principle component analysis using a minimum noise factor transform (PCA-MNF) was used to maximize the clarity of the features and to reduce the presence of noise in the eigenimages. The combination of HSI and MA-XRF (Fig. 22b) clearly showed the complementary nature of the two modalities, allowing to visualize the overall shape and several details of the female figure in the areas overpainted with lead white rich paints. Moreover, the transmission mode NIR and reflectance spectroscopic data enabled the identification of a number of pigments, supported by the results obtained with macroscopic and/or handheld XRF.

Fig. 22

R. Magritte Le portrait:a rotated photograph, b composite image obtained by overlaying the first eigenimage of MNF of 1000–1550 nm in transmission mode with Cr (light blue), Fe (red), and Hg (also red) MA-XRF maps, making visible the under painted female figure. Modified from [159]

Similar results were obtained by combining MS-IRR and HS-RIS and SR-XRF mapping to analyse the painting The Blue Room (1901) by Picasso, uncovering the existence of a hidden male portrait. [160] IRR was performed with an InGaAs camera and liquid nitrogen-cooled analog InSb to cover range up to 2450 nm. For the HS-RIS, an optimized whiskbroom scanner with EMCCD and an InGaAs array to cover respectively range in 380–1000 nm and 967–1680 nm. The tandem use of micro-analytical methods, RIS, and SR-XRF, revealed important information about the early Blue period painting and the untitled portrait beneath it, that would have been inaccessible to any one of the techniques applied separately. In fact, it was underlined that no individual technique could have completely characterized Picasso’s palette or reconstruct the hidden portrait, mainly due to similarities in the materials used for the two superimposed compositions. In some cases, the absence of an intermediary ground layer made difficult the interpretation of the RIS results, while the zinc white pigment layer throughout the surface hampered in part the detection of SR-XRF signal from the underlying painting.

In 2017, three complementary non-invasive techniques—i.e. HSI, MA-XRF, and Raman spectroscopy – enabled a comprehensive study of the The violin player (1922) by Van Dongen. [161] HSI measurements were performed with apushbroom scanner equipped with Specim camera covering 1000–2500 nm. The combined analysis facilitated not only the characterization of the pigments palette, but also the identification of the painting materials used for realizing details not visible to the naked eye, allowing to uncover a hidden portrait of a woman under the painting. Such portrait resulted especially visible in the 2250–2500 nm spectral range, showing some stylistic features similar to another Dongen’s portraits. MA-XRF mapping, while allowing for the identification of pigments in combination with Raman spectroscopy, provided the XRF elemental map of zinc, which turned particularly relevant in highlighting the traits of the hidden female portrait, as well as revealing its composition based of zinc oxide pigments.

The pushbroom Resonon Pika II HSI system (400–900 nm), portable MA-XRF, together with SR-XRF results were post-processed jointly to reveal the biography of a degraded manuscript recycled as binding material for a 16$^{th}$ century printed edition of Hesiod’s Works and Days [162]. The information contained in the datasets acquired on site were fused using a machine learning process for an enhanced analytical result. This method enabled to visualize the hidden text, as well as identify pigments and inks proposing thus an innovative method for improving imaging of hidden text using portable devices.

A year later, another 15$^{th}$ century illuminated manuscript was investigated with a combination of in situ non-invasive techniques, namely XRF mapping and point analysis, and VIS–NIR pushbroom Specim (Oulu, Finland) HSI camera [163]. The VIS–NIR HS system allows for a 400–1000 nm coverage with 212 wavelength channels with a spectral sampling of 2.8 nm and 1600 spatial pixels. Depending on the objectives, spatial sampling of 250 $\mu$m or 40 $\mu$m could be achieved. Semi-automatic HS data treatment with MNF-PCA transform and SCM provided an overview of the pigments distribution. The combined approach led to unprecedented information on the manuscript, providing new clues for its attribution by revealing hidden iconographic features, as well as describing the illuminator’s palette and techniques in detail.

In 2017, IRR was complemented with XRR and MA-XRF scanning for assessing the authenticity of a painting attributed to a young Van Dyck [164]. The authors employed Osiris IR camera (Opus Instruments, UK) equipped with an InGaAs array operating in 900–1700 nm to record six reflectograms stitched together in Photomerge in Adobe Photoshop CS5. Data provided by the multi-analytical approach enabled to highlight technical details, which were considered atypical for Van Dyck’s early period. Main observed points opposing to the authenticity of the painting were the average quality of the support, the reddish ground layers, the absence of an iron-based earth pigment in the fluid sketching medium and the usage of the blue pigment smalt in the drapery (instead of indigo). Nevertheless, the authors did not exclude a seventeenth-century origin of the painting, since materials and painting technique did not depart in any significant way from the standard practice of that period.

In 2019, comprehensive analysis of underdrawings and pentimenti in a painting by Sacchi was performed using MS IRR and XRR. [165] A modified reflex (NIR converted NikonD7100 with 1 IR cut and 3 long pass filters) and TE cooled InGaAs cameras (XENICS Xeva1.7640”) provided for IRR in VIS-NIR (370–1700 nm). The images were registered in ImageJ and processed in MatLab integrated with Hypertools, a free graphical user interface for Hyperspectral Image Analysis. The authors highlighted the limitation of the univariate versus a more exhaustive multivariate approach. Post-processing of RIS images, including non-conventional false-colour images and chemometric methods, revealed important spectral features not discernible in raw data. Especially PCA proved crucial in underscoring spectral variances, confirmed also by radiographic images.

Recently, Tserevelakis et al. [166] has focused on overcoming the limits of RIS in the visualization of hidden features, due to highly absorbing/opaque over-paint layers, by testing the performance of the cutting-edge photoacoustic imaging (PAI). The method, widely employed in biomedical research, is based on the photoacoustic effect, which exploits the presence of opaque media inside the painting. In fact, when the light from a pulsed or intensity modulated source, irradiating the painted object from its backside, is absorbed by the materials (such as those generally used for underdrawings), ultrasonic acoustic waves are generated. Acoustic signal can be collected from the other side of the painting, thus revealing the spatial distribution of the absorbing components. The method was tested on a series of mock-up oil paint samples with underdrawings made of different materials, whose absorption properties were previously analysed by UV–VIS-NIR spectrophotometry. Then, the output of PAI was evaluated in comparison with NIR multispectral imaging. Noteworthy, photoacoustic techniques not only resulted effective in providing the visualization of underpaintings, but also enabled the 3D survey of the superimposed painting layers in the modality of signal attenuation (PAcSAI) imaging. Thickness measurements were based on the frequency analysis of the transmitted photoacoustic waves, undergoing an exponential attenuation effect as they propagate through the material [167].

5.3 3D and cross-sectional survey

A number of studies have illustrated the advantages of combining RIS with other optical techniques for the morphological and cross-sectional survey of paintings. First examples report on the integration of visible colour, IR, UV and 3D data, for assessing the conservation state of a 16$^{th}$ century panel painting attributed to Leonardo da Vinci [168] and a panel painting by Antonio Pisano [169]. Shape measurements were carried out using a high-resolution laser scanning micro-profilometer, whereas 2D data were provided by UV fluorescence multispectral image acquisition system and a scanning device for simultaneous RGB colour imaging and IR reflectography. The registration and merging of 2D images with the micrometric 3D map of the surface allowed to precisely locate and quantify the presence of overpaintings, detachments or abrasions, as well as to evaluate the painting’s conservation state in view of any conservation treatments.

A few years later, Optical Coherence Tomography (OCT), a low coherence interferometric technique originally applied in ophthalmology, was introduced for the cross sectional analysis of painted objects. [170, 171] In 2011, MS RIS results, obtained with the PRISMS device (Portable Remote Imaging System for Multispectral Scanning), were successfully integrated with those provided by OCT. [172] The PRISM was specifically designed for in situ, simultaneous high-resolution spectral and 3D topographic imaging of large painted surfaces, specifically wall paintings. Imaging could be performed at sub-mm resolution both at close range and remote distances (up to 10 m), by using a lens or a small telescope, respectively. The collection system was equipped with 10 interference filters in the 400–900 nm with bandwidth of 40 nm (the 880 nm filter has a bandwidth of 70 nm). In the short wave infrared (SWIR) range of 900–1700 nm, an AOTF spectrograph, tunable to any wavelenght with adjustable bandwindth, was used along with an InGaAs camera. The idea of combining the two techniques came from the need to merge spectral information with cross-sectional images of the subsurface microstructures. In this manner, colour information and pigments identification could be easily derived from spectral reflectance measurements, whereas the scattering and absorption properties of paints investigated by OCT could give additional information for pigments’ identification. Furthermore, OCT probing of the amount of scattering within the varnish layer provided an indication of the state of degradation of the varnish.

In 2013, Liang et al. [173] investigated the best spectral window for the en-face imaging of subsurface materials in painting with IRR and OCT. Based on the spectral properties of a set of historic artists’ pigments/paints, the 2.2 $\mu$m wavelength turned out to be the most suitable for visualizing preparatory sketches, both for direct infrared imaging and OCT, suggesting that broadband sources at around 2 $\mu$m are highly desirable for OCT applications in art and in material science in general.

Further implementation of the PRISMS device [174] proved effective in providing images at transverse resolutions of tens of microns remotely (from distances up to 35 m), making high-resolution imaging possible from the ground for areas at hardly accessible heights. The proposed scanning system bypassed the need of additional devices for distance measurement, being the focusing distance a by-product of spectral imaging, achieving a distance accuracy of a few millimetres on wall painting targets at distances of around 10 m. Thanks to the capability of PRISM to register simultaneously 3D shape, spectral reflectance and colour images, extra processing procedures for the co-registration of 3D data and spectral images could as well be avoided. The performance of the device was tested for the large-scale recording of quantitative data and detailed examination of specific parts of a wall painting in a Buddhist cave temple from the 10$^{th}$–11$^{th}$ century, capturing images at the ground level of a patch of the ceiling. The colour image, derived from the MS images, was then automatically stitched into a seamless mosaic using a cross-correlation routine. [174]

The combination of RIS and 3D survey proved especially useful for the documentation and monitoring of restoring operations, which may cause irreversible chromatic and morphological variations - e.g. the cleaning process. It was demonstrated that such changes could be monitored on several types of painted artworks, through the joint application of non-invasive optical technique, including MS RIS, OCT and laser scanning micro-profilometry [175, 176]. On the one hand, OCT provided the visualisation of the painting stratigraphy, thus allowing for a quantification of the patina thickness before and after the cleaning. On the other hand, the superficial morphology of the painting surface was analysed through scanning micro-profilometry and displayed by means of topographic maps, highlighting variations in roughness due to the removal of the material.

Later, Targowski et al. [177] illustrated the effectiveness of combining VIS-NIR reflectance and UV fluorescence imaging with OCT, to provide non-invasively stratigraphic information on the painting The Landsdowne Virgin of the Yarnwinder by Leonardo. OCT tomograms, in the form of both cross-sections and en-face images, were software-registered onto the relevant areas of high resolution visible, UV-induced fluorescence and NIR images. The ambiguities in interpreting features observed by UV-induced fluorescence as alterations was overcome by integration of OCT data, revealing the structure of the painting’s superficial layers and quantifying and discriminating between varnish layers and retouchings.

The successful integration of 2D and 3D data for the acquisition of iconographic, morphological, structural, and compositional information on two artworks possibly linked by the same iconographic subject—i.e., a panel painting by Rubens and a tapestry by Févère—was shown by Dal Fovo et al. [154]. Three-dimensional mapping of the painting surface allowed for the quantification of warps and deformations characterizing the wooden support and affecting its current appearance. Then, 3D coloured models of selected areas of the tapestry were obtained by merging the topographic maps and RGB images generated by the multi-spectral scanner for exact measurement and correlation of 3D features of the warp threads with the RGB images (Fig. 16).

Another way to obtain 3D models of artworks for conservative purposes is based on combined application of reflectance transformation imaging (RTI) and shearography [178]. The first technique is a digital acquisition process that captures a set of images of a subject from a single view under varying lighting conditions: given a static object and a fixed camera view, view-dependent per-pixel reflectance functions are modelled from captured data [179]. The second is a full-field speckle interferometric technique for determination of surface displacement derivatives, suitable for measurement of the object’s shape, analysis of vibrational modes, as well as measurements of curvature and twist [180]. In [178], a customized shearography system, synchronized with an IR camera to provide thermal maps, was tested for temporal characterization of strains that occur on canvas paintings when subjected to changes in exhibition conditions. The detection capabilities of the proposed shearographic system were evaluated on a painting by comparison with gradients of displacement of surface topology obtained by reflectance transformation imaging. It was demonstrated that damage or alteration in the layered structure could be clearly located, providing insight into the time-based thermodynamics of painted surfaces by mapping the actual magnitude and direction of displacements. Furthermore, the integration with RTI enabled to correlate the detected discontinuities with morphological features visible on the paint surface. Results reported in Fig. 23 shows a clear correspondence between the strong gradient of displacement of the paint surface provided by shearographv and the presence of a high spatial slope indicative of a crack highlighted by RTI.

Fig. 23

Oil on canvas painting, unknown artist, private collection:a the area measured by shearography and RTI (dashed rectangle); b side-by-side comparison of a shearography-generated gradient of displacement map with ghost overlay, c RTI-generated views of the same paint surface as seen with specular enhancement, d and as a contrast-enhanced surface normal map. Modified from [178]

Alternative 3D methods were introduced for the cross-sectional survey of paintings, when near infrared imaging was found not successful in visualizing inner features and stratigraphy in paintings, due to the presence of superimposed opaque layers [181, 182]. Among others, terahertz (THz) imaging and THz time-domain spectroscopy (TDS) were tested for visualizing and measuring the substructure of paintings on canvas and drawings [183]. These techniques were applied in comparison with the more traditional IRR and XRR valuate their effectiveness [184,185,186]). In [186], measurements were carried out on a mock-up painting on canvas consisting of several stripes of raw umber drawing covered with lead white. Although the hidden paint strokes were vaguely revealed by RIS from the canvas support side of the painting (Fig. 24c), THz imaging provided information on the thickness of the hidden paint layers (Fig. 24a). Specifically, by monitoring the time delay between reflections off different inner layers, it was possible to measure their optical thickness, thus obtaining the in-depth survey of the painting. Moreover, it was observed that, contrary to THz-rays, X-rays did not penetrate the lead white paint and, thus, could not be used to image the underpainting (Fig. 24b). The study demonstrated that, in presence of highly opaque overpainting, THz-TDS imaging may enable the visualization of hidden painting layers (even if at lower spatial resolution than IRR), as well as providing 3D information, such as paint layer thickness.

Fig. 24

Paint sample: a time separation between the 2$^{nd}$ and the 3$^{rd}$ positive peak in the reflected THz signal, as a function of position on the canvas, which is a measure of the optical thickness of the raw umber strokes, b X-ray transmission image: the presence of the lead white paint obstructs the penetration of X-ray, hampering the visualization of the underpainting, c IRR allows to identify the raw umber strokes, which are visualized through the canvas but with a limited contrast. Modified from [186]

Further combined VIS–IR HS RIS and THz imaging investigations carried out on a 17$^{th}$ century fresco painting on tavella by Gherardini [187]. The HS data made it possible to determine the state of conservation, revealing the presence of several cracks on the paint layers and in the outer part of the plaster, as well as making visible traces of inhomogeneous underdrawings. Furthermore, from the reflectance features observed in the 400–900 and 950–1650 nm spectral ranges, it was possible to identify the presence of smalt, a cobalt-based silicate glass, and other pigments, such as natural and burnt Sienna and umber earths. Furthermore, reflection imaging performed at 97  GHz provided a qualitative evaluation of the position of defects inside the tavella.

6 Future perspectives

Nowadays, a significant number of scientific articles are published annually on HS and MS imaging for various applications in heritage science field. RIS technique is for its intrinsic characteristics expected to become an indispensable method in the heritage science domain. RIS is a complex multidisciplinary method defined as the simultaneous acquisition of spatial images in spectrally contiguous bands. Rather than collecting a single spectrum at a single site, as in spectroscopy, HIS records a spectral volume containing a complete spectrum for every measured pixel. The so-obtained chemical images allow for the visualization of different chemical constituents based on their spectral signatures. The spectral specificity enables to detect and distinguish many different compounds in spite of their similar colour, morphology or overlapping spectra. The build-up of the chemical maps permits the identification of different entities simultaneously. Its main value lies in the ability to spatially resolve heterogeneous samples, provided that the materials exhibit characteristic spectral features and are present at a minimum concentration or converge in a pixel. To reduce the huge amount of data generated by HIS, data reduction algorithms to select specific wavelengths and/or region of interest are implemented. Measurements throughout the visible, near-infrared and shortwave regions of the electromagnetic spectrum make RIS a powerful technique for characterizing artwork materials and the trend is to stretch further the spectral range to encompass as broad spectral interval as possible.

Key advantage of RIS is the integration of spectroscopic and imaging approach for disclosing hidden features, for differentiation/classification of various components and their spatial distribution in a noninvasive way. Indeed, recent significant improvements in instrumentation for NIR spectroscopy, with enhanced detection sensitivity, lowered background noise and extended spectral range, lead to the increased availability of commercial devices. Moreover, several manufacturers have specialized in producing the entire hyperspectral imaging units. RIS sensors combined to chemometric software tools (in-line soft sensors) are expected to be routinely applied for manufacturing process understanding and control. Other strategies could foresee the implementation of various machine learning algorithms for classification, pattern recognition and prediction derived from existing data. Future projects may be also oriented towards the development of systems for screening of large and non-planar cultural assets where traditional whisk- or push- broom scanners may face experimental constraints.

RIS method alone has robust but not exhaustive capacity for the complete characterisation of the composite painted systems constituting artworks. However, RIS can be easily combined with other spectroscopic or non-spectroscopic sensors to measure a larger variety of properties. This trend will advance and, as a consequence, data complexity will further grow through combination of multiple and multivariate sensors. Advanced methods of data analysis will necessarily need to be developed for proper data treatment in order to draw the correct conclusions. Due to the increased association of multimodal sensors, another rapidly evolving field will concern on one hand the development of scanners producing already registered (multimodal) datasets and on the other hand data fusion methodologies. It is likely that these two lines will proceed in parallel.