Keywords

1 Introduction

Metal soap protrusions are a form of deterioration that affects scores of oil-based paintings made from antiquity until the present day (Noble et al. 2002, 2005; Higgitt et al. 2003; Keune 2005; Jones et al. 2007; Ferreira et al. 2011). Understanding how metal soap protrusions form and develop over time constitutes a major challenge to painting conservation, as this widespread phenomenon is known to alter the visual appearance of the painted surface (Noble et al. 2005; Noble and Boon 2007; Shimadzu and Van den Berg 2006; Centeno and Mahon 2009) and to compromise its chemical and mechanical stability (Rogala et al. 2010; Maines et al. 2011). A growing body of literature has emerged on the investigation of metal soap protrusions in paintings over the past 20 years. While most of this research has focused on investigating metal soap protrusions at the micro- and molecular scale using a variety of techniques requiring micro-samples (Heeren et al. 1999; Noble et al. 2002; van der Weerd 2002; Higgitt et al. 2003; Plater et al. 2003; Keune 2005; Osmond et al. 2005, 2012, Keune et al. 2005; Keune and Boon 2007; Cotte et al. 2007; Spring et al. 2008; Ferreira et al. 2011, 2015; Osmond 2014), in this study, an easy-to-implement and non-invasive methodological approach is described, using artworks by Georgia O’Keeffe as models, to document the macro distribution of protrusions.

Based on a survey initiated by the Georgia O’Keeffe Museum in 2009, a subset of O’Keeffe’s paintings produced between 1920 and 1950 were identified as having disfiguring micro-protrusions scattered across their surfaces. These protrusions exhibit a strong UV-induced fluorescence response, range in size from 10 to greater than 200 μm, and occasionally appear erupted with a “caldera-like” shape consistent with soap aggregates found in other modern and early modern paintings (Faubel et al. 2011; O'Donoghue et al. 2006; Osmond et al. 2005; Ferreira et al. 2011; Duffy et al. 2014; Helwig et al. 2014). The protrusion formation process must have started at an early stage in these artworks’ history: in a 1947 correspondence between O’Keeffe and conservator Caroline Keck (1947), the artist mentioned that she noticed opaque, granular textures and pinpoint losses appearing in several of her oil paintings created between 1928 and 1936, some of which today exhibits particularly large protrusions.Footnote 1 The identification of these protrusions in the 2009 survey led to a number of questions about when they first developed on these artworks and whether they were still actively changing.

The presence and potential ongoing growth of protrusions on paintings raise significant concerns for their long-term preservation. Characterizing the protrusions, their distribution over the surface of the artworks and monitoring their development in time have now become a conservation priority not only for O’Keeffe’s oeuvre but for all paintings affected by protrusions. Likewise, investigating the mechanisms and factors promoting the protrusion development in artworks has become essential, especially since the collections are often used in traveling exhibitions and are subjected to environmental fluctuations that may contribute to the ongoing development of protrusions.

In this work, in order to document the dynamic development of protrusions over time, a novel photometric stereo (PS) protocol was developed to image the 3D-texture of the painted surface where fine features of ∼10 μm wide can be resolved. We achieve these figures of merit by exploiting the fluorescence properties of pigments and binding media when excited by ultraviolet wavelengths. This isotropic response (Rost 1995) is ideally suited to using PS, which assumes Lambertian (diffuse) reflectance from a surface (Woodham 1978). As a second step, we also describe two data processing methods for detection of the protrusions using the PS data as an input: a manual count is compared to a data-driven approach. These counts of protrusions in different regions and the measurement of their size distribution are then correlated with paint composition.

2 Experimental Capture and Processing Procedures

The Georgia O’Keeffe oil painting Pedernal (Fig. 22.1a) has been chosen as a case study to illustrate the processing and analysis of PS data. Imaging was undertaken on a region of the painting, demarcated in Fig. 22.1a that exhibits a high number of protrusions and is representative of the overall palette. The region size of approximately 7.6 × 5.0 cm (height × width) was chosen to achieve high spatial resolution of ~10 μm using our camera/lens configuration. Specifically, in the conditions used, spatial sampling of ~15 μm per pixel over a field of view of 4 cm was achieved. It should also be noted that exposure to these UV wavelengths was kept at a minimum, similar to that needed for a UV photograph of a painted surface, to ensure the safety of the painting. The setup utilizes a 365 nm UV lamp (fitted with a Sylvania F15 T8/BLB bulb) as the illuminant and a Canon EOS 5D Mark III camera equipped with a 50 mm prime lens.

Fig. 22.1
figure 1

Preprocessing pipeline of photometric stereo data illustrated for case study (a) Georgia O’Keeffe unvarnished oil painting Pedernal (1941–1942, 1997.5.12, CR1029, 51.1 × 76.8 cm, Georgia O’Keeffe museum, Santa Fe, NM). Region of ca. 7.6 × 5.0 cm, demarcated by yellow box, was selected as representative example for investigation. A single image illuminated with a directional light (b) was fit with an orthogonal Legendre polynomial to calculate a map of light drop-off (c) which shows the light originating from the direction of the lower left corner of the image. To achieve a precise measure of the lighting direction, polynomial-fit image was reshaped into polar coordinates (top d) to determine the angle θ of the illumination vector L. Cartesian and polar coordinates of illumination vector L are schematically represented (bottom d). The xL, yL, zL position of each illumination direction (e) was determined using an optimization procedure described in the text. PS was calculated using linear least squares to produce a normal vector map (f). The white box demarcates the detail region shown in Fig. 22.2

Full size image

2.1 Photometric Stereo Data Acquisition and Processing

2.1.1 Capture

At its core, our method utilizes a common camera configuration described previously for PS (Woodham 1978) and reflectance transformation imaging (RTI) measurements; the camera is fixed orthogonal to the artwork and a series of photographs are taken with different angles of illumination (Malzbender et al. 2001; Padfield et al. 2005; Mudge et al. 2005, 2006, 2008; Fattal et al. 2007; Earl et al. 2011, 2010a, 2010b; Malesevic et al. 2013;Duffy et al. 2013). Following the well-known photometric stereo equation, albedo (color) and normal maps (shape) of the object surface are recovered:

$$ I=\overset{\sim }{\boldsymbol{N}}\cdotp \boldsymbol{L},\kern0.75em \overset{\sim }{\boldsymbol{N}}=k\boldsymbol{N}, $$
(22.1)

In this parameterization, the albedo (color) k is absorbed into the surface normal so that \( \left|\overset{\sim }{\boldsymbol{N}}\right|=k \). Also the image intensity is given by I(x, y) under illumination vector L.

Shape recovery in PS requires that the surfaces being measured scatter light diffusely, often called Lambertian scattering. However, few materials exhibit ideal diffuse behavior but instead reflect light specularly. Painted surfaces often exhibit both diffuse and specular properties. These non-ideal surfaces are difficult to model accurately using PS alone (Ma et al. 2007), so recent studies have highlighted the advantages of using fluorescence-based PS for 3D-shape reconstruction compared to previous methods using reflected light (Sato et al. 2012; Treibitz et al. 2012). Stokes fluorescence emissions induced by a directional short wavelength light are isotropically radiated at longer wavelengths with ideal Lambertian characteristics where the emission intensity varies according to the cosine of the lighting angle (Kratohvil et al. 1978; Gordon et al. 1993; Glassner 1995). Therefore the use of fluorescence emission as the source of light for PS of painted surfaces can enable more accurate, precise, and repeatable shape measurements than the use of reflected light.

2.1.2 Preprocessing

2.1.2.1 Calculating the Illumination Vector

In a typical “free-form” PS or reflectance transformation imaging capture setup, a reflective ball is used to record the position of a far-light source L (Mudge et al. 2008). Here we have eliminated the need for using the reflective ball through an algorithm designed for near-lighting conditions (Huang et al. 2015; Cossairt et al. 2015). One of the principal benefits of removing the mirror ball and other calibration hardware from the scene is that the entire field of view may be dedicated to capturing the desired image. Also, this method where the light position is estimated directly from the object itself exploiting the inverse square law of light drop-off may be more accurate than using a mirror ball (Huang et al. 2015). This is especially true if the material being measured is a true Lambertian reflector as is the case of fluorescence measurements (Treibitz et al. 2012). Lastly, the weak intensity of most fluorescent sources necessitates near-light conditions in order to provide enough photon flux to produce an acceptable signal to noise characteristic in the captured images. In our example, the light was positioned ~500 cm away from the surface.

In comparison to Huang et al. (2015), in this study, a different image processing pipeline is used which is more user-friendly for conservators, museum imaging departments, and conservation scientists interested in adopting the framework described. All of these steps were performed with either open-source and off-the-shelf software, such as ImageJ (Schneider et al. 2012, Schindelin et al. 2015). The code used in this project is available from an online repository (Github) as Python scripts that can be readily downloaded and installed into ImageJ.Footnote 2

Light position estimation was achieved in a three-step process, as illustrated in Fig. 22.1b–f. First, each lighting direction image (Fig. 22.1b) was fit with an orthogonal polynomial in ImageJ, as exemplified in Fig. 22.1c, that is normalized to the mean grey value. While polynomial fitting of an image is typically used to compensate for uneven illumination, here we use the fit image (Fig. 22.1c) to estimate illumination direction. The fit-image can be reshaped into polar coordinates by radially slicing the image (Fig. 22.1d), from which the angle of illumination θ can be directly extracted from the point of maximum intensity. Second, the polynomial image is used to normalize or “flat-field” each lighting direction image to correct for the unevenness of illumination induced by the near light. Third, we borrow from the near-light model (Huang et al. 2015), in which the Cartesian coordinates (xL, yL, zL) of the vector of illumination L are calculated from the direction and magnitude of the light drop-off. It is assumed that all light source positions are equidistant to the center of the region of investigation (Fig. 22.1d), meaning |L| = 1.

As illustrated in Fig. 22.1e, each lighting direction L is determined by iteratively adjusting a parameter m while updating the surface normal \( \overset{\sim }{\boldsymbol{N}} \) . This is performed until the difference between \( \boldsymbol{L}(m)\cdotp \overset{\sim }{\boldsymbol{N}} \) and I becomes very small as per this cost function:

$$ \mathrm{argmin}_{\overset{\sim }{\boldsymbol{N}},m}\big|\boldsymbol{L}(m)\cdotp \overset{\sim }{\boldsymbol{N}}-I\big|\kern0.5em $$
(22.2)
2.1.2.2 Calculating Surface Normal, Depth, and Albedo Maps

Based on the estimated vectors of illumination L, the PS Eq. (22.1) is solved using least squares to recover the albedo and surface normal maps, the latter shown in Fig. 22.1f. Examples of x and y gradients, depth and albedo images are illustrated in Fig. 22.2a–d as high-resolution details from Pedernal. The depth map (Fig. 22.2c) is calculated by integrating the x and y gradients using standard algorithms such as described by Frankot and Chellappa (1988). Clearly apparent in the center of the image is a large protrusion (200 μm across) with other smaller protrusions (30–150 μm) peppered across the surface that have whitish to yellowish fluorescence (Fig. 22.2d). Figure 22.2e illustrates the 3D visualization of the depth map that enables characterization of the morphological features of the protrusions.

Fig. 22.2
figure 2

After the flat-fielding procedure described by Fig. 22.1, the complete set of raking light images were used to calculate normal vector, depth, and albedo images. Details (a–d) of an area of 2.3 × 2.3 mm show the x-gradient (a) and y-gradient (b) of the surface normal vector N, the depth map (c) – where the grey scale intensity corresponds to relative height – and albedo (d) providing information on the diffuse color generated by the UV-induced fluorescence of the surface with underlying contrast from shape removed. The 3D-rendering (e) shows a “caldera” shape of the central protrusion

Full size image

2.1.3 Protrusion Detection

The use of UV-induced fluorescence presents two benefits in addition to the improved accuracy of the PS measurements. First, the calculated albedo k is a perfectly diffuse and flat record of fluorescence which aids in the accurate detection and segmentation of the protrusions. Second, information on the UV-induced fluorescence response of the protrusions and of surrounding material is simultaneously collected in the visible range, which can contribute to material differentiation and characterization.

Using both the albedo and shape information, two data analysis methods were tested and compared to investigate the occurrence and distribution of protrusions in different colored areas. The first method, which is manual, uses ImageJ to detect protrusions on the albedo image with local color thresholding. The albedo image allows for a convenient visualization of the protrusions, as shown in Fig. 22.2d, due to their strong fluorescence response. This method provides quantitative information on the protrusions by detecting all visible protrusions within the investigated region. The second method is a fully automated detection scheme using an experimental algorithm implemented with Matlab and OpenCV (Bradski 2000). This algorithm has been designed for unaided detection of the protrusions, based on the combination of characteristic colors and surface features.

2.1.3.1 Manual Protrusion Detection

Visible image of the investigated region is shown in Fig. 22.3a. For each observable protrusion in the albedo image, contiguous pixels of similar RGB values within an optimized adjusted range are hand-selected to fit the outline of the protrusion area (via the “wand tool”). All selected pixels are used to create a map of the protrusion distributions illustrated as a red overlay on the image of the fluorescence response (Fig. 22.3b).

Fig. 22.3
figure 3

(a) Visible image of the investigated region from Pedernal – shown in Fig. 22.1a – used to test methods for data analysis; (b) albedo image of the same region with overlaid binary mask of protrusions distribution (in red) manually selected using ImageJ; (c) albedo image with schematic indication of the eight different colored areas for segmentation; (d) mask from the albedo image for the light red painted area, labeled 3 in Fig. 22.3c; (e) steps for protrusions analysis in ImageJ: detail of the mask of the light red area (left); results for that detail from applying the mask of the light red area to the protrusion mask: distribution of protrusions (in black) for the light red area is obtained

Full size image

The visible image (Fig. 22.3a) was manually segmented to isolate areas that appeared to have similar colors based on the visible and albedo images (Fig. 22.3c). Each of these areas was masked (Fig. 22.3d) to count the total number of protrusions per color. This, together with their location, size, and area were all recorded (Fig. 22.3e). The process was repeated for each segmented color area.

Despite being time-consuming, the manual method is advantageous since it is very easy to implement and accurate for detection of protrusions larger than 10 μm, considering the resolution of the PS data used in the current example.

2.1.3.2 Automated Protrusion Detection

A custom algorithm was applied to test automated quantification of the protrusions in the images acquired by PS. As for the manual method, each color area was manually segmented and masked (Fig. 22.3c–d). For the automated detection algorithm, the masks were then applied to both the albedo and normal images. Color thresholding and template matching on the albedo image were used to generate a binary mask, where a value of 1 indicates the presence of a protrusion. The normal image was integrated using the Frankot-Chellappa algorithm to produce a depth/height map of the surface of the painting. The OpenCV blob detection routine was then used to locate protrusions in the depth image, generating a second binary mask. A composite mask is then generated from the union of the two binary masks. The same procedure for analyzing protrusion statistics within each color region is then applied. The combination of information from the color albedo and surface-shaped images enables better rejection of false-positive protrusion detections, such as those that may be caused by brushwork texture or areas where the primer is apparent.

For both methods, the number of protrusions/cm2 was calculated as a function of protrusion width for each color area (Fig. 22.4a).

Fig. 22.4
figure 4

Results of protrusion analysis and comparison of methods: (a) results of “ground truth” manual method of analysis. Histograms of protrusion occurrence and range of size for the eight colored areas; (b) results from the computer-automated method for the light red region: mask (top); details of results of protrusion detection (circled in red) overlaid onto depth map (bottom left) and albedo map (bottom right); the manually detected protrusions missed by automated processing are circled in yellow for comparison; (c) table comparing the occurrence of protrusions detected (number/cm2) for each colored area in the investigated region by the two data analysis methods. Colored areas with high number of protrusions are shown in blue

Full size image

2.2 X-Ray Fluorescence (XRF) and Portable Fourier Transform Infrared Spectroscopy (FTIR)

XRF imaging was carried out on the same region captured during the PS analysis (Fig. 22.1a), using an ELIO X-ray fluorescence imaging spectrometer (XGLab), equipped with a Rh tube and 1 mm spot size. Rastering was executed with a 250 μm step size and 1.6 s acquisition time for each point. The instrument was operated at 50 kV and 40 μA. The open access PyMCA software (Solé et al. 2007) was used to fit the XRF map and obtain the distribution of each element.

FTIR spectra were collected for each painted color area found in the region of interest using a Bruker Alpha small footprint portable FTIR spectrometer, in reflection mode and spectral range 6000–400 cm−1, a measurement spot of 6 mm in diameter, and working distance of ~15 mm. Two hundred and fifty six scans were acquired with a 4 cm−1 resolution.

3 Results and Discussion

3.1 Detection of Protrusions

3.1.1 “Ground truth” Manual Method

The red overlay on the fluorescence image (Fig. 22.3b) shows the distribution of protrusions across the painted surface. Protrusions appear most abundant in areas corresponding to light tonalities (e.g., light red, light green, beige) whereas they are almost absent in other darker tonalities (e.g., dark green, dark red). Frequency histograms (Fig. 22.4a) show the number of protrusions/cm2 as a function of their width. From these data, the beige, light green, light red, yellow, and yellow green areas have a high number of protrusions (34 to 57 protrusions/cm2). In contrast, the number of protrusions in the dark green, dark red, and orange areas is significantly lower, never exceeding 10 protrusions/cm2. As will be discussed in more detail below, these data show that lighter and darker shades of the same colors (as in dark vs. light green – likely made from the same chromium-based pigment as determined by XRF) – clearly show dissimilar protrusion counts.

The profiles of size distributions are overall similar for the different colored areas, with a dominant proportion of protrusions in the 100–200 μm range. Only the dark red and yellow green areas exhibit a slightly higher proportion of small protrusions less than 100 μm wide. It can also be noted that the areas with abundant protrusions skewed to distributions with larger protrusions – with widths often exceeding 400 μm. It should be noted that for this proof-of-concept work, relatively large bins of 100 μm have been chosen to illustrate the capabilities of the method and reduce the number of false positives; however, in the future, smaller bins, especially in the 0–300 μm range, should be considered to better characterize the variety of sizes of the protrusions.

3.1.2 Evaluation of the Automated Method

Figure 22.4b shows the detected protrusions, circled in red, for the light red area using the automated algorithm overlaid onto the depth map and albedo images. For comparison, the manually detected protrusions missed by the automated method are circled in yellow. The detection of relatively big protrusions is fairly robust, while detection of small protrusions (∼<200 μm) is not accurate. The challenge in detecting small protrusions lies in the fact that false positives are generated from the surface texture produced by the brush marks and from the white priming layer, contrasting in color with paint materials in areas where it remains visible. While automation significantly reduces processing time, further development of this preliminary version of the automated algorithm is however necessary to optimize detection accuracy so that it can be reliably applied to monitoring the variation of protrusion statistics over time.

3.2 Correlating Protrusions with Paint Composition at the Macroscale

Non-invasive FTIR and XRF analysis were performed on Pedernal as a case study to investigate how the distribution of protrusions correlates with paint materials. Many of O’Keeffe’s artworks that exhibit protrusions have been painted on a specific type of commercial pre-primed canvas. Based on this observation, it was originally speculated by the museum conservators that the priming layer could have played a role in the protrusion formation, and this hypothesis was thus a focus of our investigation. As explained below, our analysis determined that, on the contrary, the distribution of protrusions correlates with the composition and color of paint layers, and therefore the degradation phenomena observed at the surface cannot be attributed to the priming.

Reflection FTIT analysis of the bare commercial primer (Fig. 22.5a) indicates that it is composed of lead white (mixture of hydrocerussite (Pb3(CO3)2(OH)2) and cerussite (PbCO3)), barium sulfate (BaSO4), and calcium carbonate (CaCO3), based on the presence of the strong derivative band of the asymmetric CO3 2− stretching at ca. 1465 cm−1 and the characteristic stretching and bending vibrational modes of hydrocerussite (3550, 1045, 680 cm−1) and combination and overtone modes of cerussite (2413 cm−1) and of calcium carbonate (4260, 2511, 1794 cm−1) (Miliani et al. 2012). It should be noted that SEM/EDX analysis of a cross section from Pedernal clarified the relative proportions of these components, indicating that the priming is primarily composed of chalk with barium sulfate, and only minor quantities of lead compounds, which are, on the contrary, dominant in the paint layers. The sharp CH stretching at 2922 and 2846 cm−1 combined with the doublets in the 1540–1520 and 730–720 cm−1 regions, as shown in the details of Fig. 22.5a, point out toward the presence of metal carboxylates (Robinet and Corbeil 2003). Only few visible protrusions are observed on the unpainted commercial primer, and when they are detected, they tend to be small (<15 μm).

Fig. 22.5
figure 5

Correlation between the distribution of protrusions and type of painting materials using complementary non-invasive analytical techniques. (a) FTIR spectra in reflectance of the primer (in blue), of the light red area (in red), and of the dark green area (in green); (b) XRF mapping of the studied area (from left to right): visible image of the area; albedo image of the area with overlays of the protrusions in red; false color K-lines elemental map distribution of iron (in red), chromium (in green), and cadmium (in blue); temperature L-lines elemental map distribution of lead where red indicates highest concentrations and blue the lowest (while only Lα peak areas are plotted here, however the Lβ distribution was found to be similar demonstrating that there is minimal contribution to the lead intensity from the ground layer)

Full size image

In all areas analyzed by FTIR, hydrocerussite and metal soap carboxylates were identified (Fig. 22.5a). Furthermore, the asymmetric carboxylate bands exhibit similar spectral features for the different colors suggesting the presence of the same type of metal carboxylates (Robinet and Corbeil 2003; Hermans et al. 2014, 2015) in both the protrusion-rich and protrusion-poor regions. These results highlight that while metal soaps are ubiquitous throughout the paint, they do not always aggregate into protrusions. Instead, paint layers with specific pigment compositions play a decisive role in whether or not surface protrusions form.

To investigate if protrusion formation is correlated with the presence of certain pigments or paints, the elemental composition was recorded via XRF mapping in the same region as the PS measurements. Elemental distributions of iron, chromium, and cadmium may be seen in Fig. 22.5b. These data indicate that red iron oxide pigments were employed for the red areas, chromium-based green pigments for the greens, and cadmium-based pigments for the yellow and orange. The distribution map of lead indicates that lead white was likely mixed with the colored pigments in varying proportions to produce lighter tonalities. The regions that are richest in lead correlate well with the areas with a higher occurrence of surface protrusions. These observations suggest that the specific type of lead white paint formulation in tubes employed by O’Keeffe and its proportion within the paint play a critical role in the development of surface protrusions, likely related to the aggregation of lead soaps. The latter would be also consistent with the observed UV-induced fluorescence response of the protrusions as reported for lead soap (Higgitt et al. 2003; Keune 2005; Keune and Boon 2007). It is currently being investigated whether the formulation of lead white tube paints O’Keeffe consistently used throughout several decades of her long career might be responsible for the observed abundance of soap protrusions and whether the same correlation in the distribution of lead and surface protrusions is observed on other O’Keeffe paintings.

4 Conclusions

Based on the case study of Pedernal, a Georgia O’Keeffe painting exhibiting high number of protrusions, this paper presents an innovative off-the-shelf methodology, based on imaging photometric stereo acquisition under UV illumination, to characterize the 2D-distribution and occurrence of the surface protrusions at the macroscale over an entire artwork. The two detection methods described show that the automated method of analysis produces fast protrusion distribution estimates, but with decreased precision with respect to the manual method, and substantial underestimation of protrusion numbers when parameters are adjusted to avoid false positives. The developed methodology also demonstrates that the uneven distribution of protrusions can be correlated to the paint composition: in the case study of Pedernal, the findings suggest that the development of protrusions, more numerous in the lead-rich areas, is primarily related to the lead white paint tube formulation in the paint layers, rather than the commercially primed canvas (containing mostly calcium and barium compounds, with only minor amounts of lead), which was previously thought to be the culprit. As an outlook, future research will focus on developing a more robust algorithm for automated protrusion counting and on providing user-friendly platforms so that the described imaging methodology can be widely applied to following the dynamic evolution of protrusions and thus directly interrogate the myriad factors that may lead to their development.