Direct Differential Photometric Stereo Shape Recovery of Diffuse and Specular Surfaces

Abstract

Recovering the 3D shape of an object from shading is a challenging problem due to the complexity of modeling light propagation and surface reflections. Photometric Stereo (PS) is broadly considered a suitable approach for high-resolution shape recovery, but its functionality is restricted to a limited set of object surfaces and controlled lighting setup. In particular, PS models generally consider reflection from objects as purely diffuse, with specularities being regarded as a nuisance that breaks down shape reconstruction. This is a serious drawback for implementing PS approaches, since most common materials have prominent specular components. In this paper, we propose a PS model that solves the problem for both diffuse and specular components aimed at shape recovery of generic objects with the approach being independent of the albedo values thanks to the image ratio formulation used. Notably, we show that by including specularities, it is possible to solve the PS problem for a minimal number of three images using a setup with three calibrated lights and a standard industrial camera. Even if an initial separation of diffuse and specular components is still required for each input image, experimental results on synthetic and real objects demonstrate the feasibility of our approach for shape reconstruction of complex geometries.

1 Introduction

Since the seminal paper by Woodham [37], Photometric Stereo (PS) has been considered a very accurate procedure for tridimensional shape reconstruction. Despite its wide use in many applications [34, 38, 39], PS suffers from several limitations that constrain its applicability to restricted scenarios. In particular, one of the most challenging issues refers to specularity, where standard PS remains substantially inaccurate. This spans from the consideration that most works dealing with PS assume objects under observation to give exclusively diffuse reflection (Lambertian). Considering the general theory of image formation in which reflections have contributions from both diffuse and specular components [25, 36], PS approaches implicitly assume specularities are negligible with respect to the global reflection of light [15, 16]. Due to the usual sparsity of the specular component, this assumption is reasonable when several images are used (typically $>$10). However, it fails when fewer images are considered. This can limit considerably the application of PS to (near-)real-time setups where a minimal set of images (i.e., three) is necessary to optimize performance. Consequently, one would greatly benefit from a new methodology for PS shape recovery which considers such minimal set of data. For this reason, extracting information from the specular component is essential even for well-studied objects, such as ceramic objects and faces, since in these cases neglecting specular effects prevents accurate shape reconstruction as shown in Fig. 1.

Here we propose a method that, given an input image separated into two components, diffused and specular, can efficiently reconstruct the 3D shape of an object with varying albedo using only three images under different light conditions. Our approach results in improved shape reconstruction compared to standard PS methods with global reflection assumed purely diffuse. In addition, our method also shows reconstruction improvements compared to PS in which only the diffuse component separated from the specular one is considered [2].

Fig. 1

The top row shows three images of a painted ceramic cup. The bottom row shows the normal map where artifacts have been highlighted and the 3D shape reconstruction using traditional PS [37]. Bottom images show artifacts due to specular reflections

In summary, this work contributes to the state of the art with

• a methodology for reconstructing surfaces with general bidirectional reflectance distribution functions (BRDFs) by using three images given a preliminary diffuse and specular reflection separation [18, 32];

• a processing procedure aimed at using both diffuse [23] and specular components with a new mathematical formulation based on the Blinn–Phong shading model.

1.1 Related Work

Aiming at providing a self-contained overview of the PS approaches, we start by discussing methods dealing with Lambertian reflection, and then we consider works that explicitly model specular components.

Once we assume that objects reflect light diffusely, the most complex case subsists whenever no information is available about the lighting setup. In such uncalibrated scenario, both the surface properties and lighting parameters of the scene have to be computed. Regarding this problem, factorization methods have provided efficient and closed form solutions. They are based on the fact that a set of images taken from a static point of view and subject to varying lights lies in a certain subspace. In particular, the reflection components can be described with a bilinear model which is a compact representation of the given image data. Hayakawa [12] first made evident such bilinear modeling assuming a Lambertian surface and a single light source. Basri et al. [4] used a more descriptive photometric model based on a spherical harmonic representation of lighting variations. Their approach can deal with images of Lambertian objects under unknown lighting conditions. These classical methods in PS, which do not have any depth assumption of the shape, are always subject to the bas-relief ambiguity [5]. On the contrary, Shi et al. [33] performed an automatic radiometric calibration by identifying a new set of constraints that can solve the Generalized Bas-Relief (GBR) transformation. A recent work by Papadhimitri and Favaro [27] approximates the GBR parameters efficiently by taking into account the information coming from the local diffuse reflectance maxima.

Furthermore, there are several recent works that study the uncalibrated PS problem with the added difficulty of dealing with specular reflection. In particular, [19] uses more than one hundred images, limiting the shape recovery to a concave/convex ambiguity and providing an approach capable to deal with objects with uniform reflectance. Chandraker et al. [7] recover surface iso-contours from differential images by restricting the positions of the light sources to a circle around the camera axis. In this case, additional information is required such as an initial normal to determine surface normals. In addition, such differential formulation uses image equation ratios with the aim to simplify the problem eliminating the dependence on photometric invariants such as albedo.

Besides the specific limitations of the methods mentioned before, the most important drawback assuming uncalibrated setups is the number of images required. Since the image acquisition is achieved by sequentially turning on and off the light sources surrounding the static object under observation, this approach is not feasible for shape recovery of movable or deformable objects.

Robust PS has been deeply studied by Ikehata et al. [15, 16] where general isotropic surfaces have been taken into account. These approaches are based on regressions and they use tens of images for accurate shape reconstruction. The reason why this method needs several images is due to its mathematical formulation. In fact, the irradiance equation is based on diffuse reflection and specular components are considered outliers or negligible. In a similar way, [10, 17, 24] treated strong specularities and shadows as missing data and they solved for the diffuse component through a matrix completion problem. However, when using a minimal set of images (e.g., 3, or 4 as in [3]), it is likely that specular components make the bilinear regression fail.

Alternatively, calibrated PS setups offer a more efficient and reliable method for 3D shape recovery. This is due to the knowledge a priori of the parameters describing the light sources and the consequent reduction in the number of images necessary for shape reconstruction. In this framework, Higo et al. [14] proposed an energy minimization method using six parameters to be tuned and with the specular lobe parametrized as a weak extension of the Lambertian shading model. Furthermore, the method works for surfaces showing either diffuse or specular reflection. Chung and Jia [8] used at least six images. However, the boundary of the shadow must be accurately separated, since it provides constraints for calculating the normal of the surface.

In general, and for the diffuse component only, the calibrated active system setup has led to a high number of custom solutions [31] that typically require a laboratory setup and accurate calibration of the devices. Instead, Hernandez et al. [13] used a less restrictive calibrated setup with only three non-collinear colored lights in a dark room with surfaces that were also photometrically calibrated. Recently, Anderson et al. [1] have extended this approach to arbitrary colored surfaces. Using three calibrated light sources is a convenient solution that allows the implementation of real-time 3D acquisition systems by multicolor illumination.

2 Formulation of the General Model

The general model presented in this manuscript is based on fixing a camera in a three-dimensional coordinate system (Oxyz) and illuminating an object with different light sources. The camera is placed in such a way that Oxy coincides with the image plane and Oz with the optical axis.

Let $\varvec{\omega }^{i} = (\omega _1^{i},\omega _2^{i},\omega _3^{i}) = (\tilde{\varvec{\omega }}^{i},\omega _3^{i}) \in \mathbb {R}^3$ (with $\omega _3^{i} > 0$ for each i-th vector) be the unit vectors that represent the directions of the light sources. The images $I_i:\overline{{\varOmega }}\rightarrow [0,1]$ are the grayscale values of the i-th image at point (x, y) belonging to the image domain $\overline{{\varOmega }}$, see Fig. 2.

Fig. 2

The scheme explains the Blinn–Phong model for a sphere on a flat background. On the left, the grayscale image defined in the rectangle $\overline{{\varOmega }}$, sectioned through the central red line. On the right, its schematic section showing the unit vectors involved in the shading modeling. The vector $\varvec{\omega }$ is uniform for all the points in the image plane and it points toward the light source. The viewing vector $\varvec{V}$ is fixed pointing in the vertical direction. The intermediate vector $\mathbf{H}$ bisects $\mathbf{V}$ and $\varvec{\omega }$ (Color figure online)

The assumptions we consider are commonly used in the PS field. We list them as follows:

A1.:

The light propagates uniformly for each source with the direction $\varvec{\omega }^{i}$ (therefore, the light rays are parallel to each other)

A2.:

Orthographic viewing geometry

A3.:

There are no inter-reflections on the surface.

Under the assumption (A2) of orthographic projection, the visible part of the scene is a graph $z = u(x,y)$ and the unit normal to the surface at the image point corresponding to (x, y) is given by

$$\begin{aligned} \mathbf {N}(x,y) = \frac{\mathbf {n}(x,y)}{|\mathbf {n}(x,y)|} = \frac{(- \varvec{\nabla } u(x,y), 1)}{\sqrt{1 + |\varvec{\nabla } u(x,y)|^2}}, \end{aligned}$$

(1)

where $\mathbf {n}(x,y)$ is the outgoing normal vector.

We consider the image function defined by the following irradiance equation:

$$\begin{aligned} I(x,y) = R(\mathbf {N}(x,y)), \end{aligned}$$

(2)

where I(x, y) is the normalized brightness of the given gray-value image, $\mathbf {N}(x,y)$ is the unit normal to the surface at point (x, y, u(x, y)), and $R(\mathbf {N}(x,y))$ is the reflectance map giving the value of the light reflection on the surface as a function of $\mathbf {N}(x,y)$ at each point. Depending on how we describe the function R, different reflection models are determined. We will describe and use two of them. As proposed in [9], it would be useful to introduce a representation of the brightness function I(x, y) in which we can distinguish different terms representing the contribution of ambient, diffuse reflected, and specular reflected light such that

$$\begin{aligned} I(x,y) = k_A A(x,y) + k_D D(x,y) + k_S S(x,y), \end{aligned}$$

(3)

where A(x, y), D(x, y), and S(x, y) are the above-mentioned components, and $k_A$, $k_D$, and $k_S$ indicate the percentages of these components, respectively, such that their sum is equal to 1. In this paper, we consider the Lambertian model for the diffuse component D(x, y) and the so-called Blinn–Phong model for the specular component S(x, y) [6]. We remark that even if the Blinn–Phong shading model is not physically based, a recent evaluation [26] shows that it provides good specular shading results compared to other physically based models. Finally, throughout the paper, we neglect the ambient component by setting $k_A = 0$. We start with a brief description of these models.

2.1 Lambertian Model

By definition, a Lambertian surface is a purely diffuse reflector and, consequently, the specular component does not exist. So, the general Eq. (3) becomes

$$\begin{aligned} I(x,y) = k_D D(x,y), \end{aligned}$$

(4)

whose diffuse component D(x, y) is

$$\begin{aligned} D(x,y) = \rho _D(x,y) \, \mathbf {N}(x,y) \cdot \varvec{\omega }, \end{aligned}$$

(5)

where $\rho _D(x,y)$ indicates the albedo of the diffuse part, i.e., the diffuse reflectivity or reflecting power of a surface. In other words, the albedo consists of the ratio of reflected radiation from the surface to incident radiation upon it. Its dimensionless nature is expressed as a percentage and it is measured on a scale from 0 for no reflection (a perfectly black surface) to 1 for a perfect reflection for a white surface. Recalling that the sum $k_A + k_D + k_S$ must be equal to 1, for a Lambertian surface $k_D = 1$ and this parameter can be omitted. In this way, the irradiance Eq. (2) can be rewritten as follows:

$$\begin{aligned} I_i(x,y) = \rho _D(x,y) \, \mathbf {N}(x,y) \cdot {{\varvec{\omega }^i}}, \end{aligned}$$

(6)

for each image $I_i(x,y)$ obtained by lighting up the surface u using the i-th light source ${\varvec{\omega }^i}$. The orthogonal PS problem consists in determining the function $u:\overline{{\varOmega }}\rightarrow \mathbb {R}$ that satisfies Eq. (6), where the unit vectors $\varvec{\omega }^{i}$ and the functions $I_i(x,y)$ are the only quantities known in the problem.

In this model, we can note that the measured light in each image only depends on the scalar product between $\mathbf {N}(x,y)$ and $\varvec{\omega }^i$ and the parameter $\rho _D(x,y)$, which describes the physical properties of the surface reflection.

In order to solve our problem, let us first fix the number of images (i.e., light sources) to $n=2$, and let us recall from [20] that one can solve the PS problem for Lambertian surfaces considering the following linear differential problem:

$$\begin{aligned} \left\{ \begin{array}{l@{\quad }l} \displaystyle \mathbf{b}_D(x,y)\cdot \varvec{\nabla } u(x,y)=f_D(x,y) &{} \quad \forall (x,y)\in {\varOmega }, \\ \displaystyle u(x,y)=g(x,y) &{} \quad \forall (x,y)\in \partial {\varOmega }, \end{array} \right. \end{aligned}$$

(7)

where $\mathbf{b}_D$ is a diffuse 2D vector field defined in $\overline{{\varOmega }}$ by

$$\begin{aligned} \mathbf{b}_D(x,y)=D_2(x,y)\tilde{\varvec{\omega }}'-D_1(x,y)\tilde{\varvec{\omega }}'', \end{aligned}$$

(8)

with $\tilde{\varvec{\omega }}' = (\omega '_1, \omega '_2)$ and $\tilde{\varvec{\omega }}'' = (\omega ''_1, \omega ''_2)$, and

$$\begin{aligned} f_D(x,y)=D_2(x,y)\omega '_3-D_1(x,y)\omega ''_3. \end{aligned}$$

(9)

From now, in order to avoid confusion between components of the same vector and different vectors, we will use the following notation: $','','''$ as superscripts will denote three different vectors (e.g., $\varvec{\omega }', \varvec{\omega }'', \varvec{\omega }'''$), the numbers 1, 2, 3 as subscripts will indicate the components of a vector (e.g., $\varvec{\omega }' = (\omega '_1,\omega '_2, \omega '_3)$).

Note that the problem in Eq. (7) is solved regardless of the albedo values thus providing an approach that can model objects with varying material properties.

Even if the differential problem (7) has a unique solution, the need of the boundary condition g(x, y), which is unknown in our case, obliges us to use a third image. Section 5 will explain in practice how to use the information coming from three diffuse available pixels and how to use (7) when the diffuse component is corrupted (e.g., specularities).

2.2 Blinn–Phong Specular Model

There are different models that account for specular reflections. In this paper we consider one of the most popular ones, the Blinn–Phong model [6]. As graphically explained in Fig. 2, it is a modification of the Phong model [28]. Briefly, specularities are modeled based on the intermediate vector $\mathbf {H}$ that bisects the angle between the unit vectors $\varvec{\omega }$ and $\mathbf {V}$. By using this model, it is possible to produce a faster algorithm in terms of CPU time when both observer and light source are placed at infinity because $\mathbf {H}$ is independent of the position and orientation of the surface.

For this model, the specular component related to the i-th image $I_i(x,y)$ is defined as follows:

$$\begin{aligned} S_i(x,y)= \rho _S(x,y)\left( {\mathbf{H}^i}\cdot \mathbf{N}(x,y)\right) ^c, \end{aligned}$$

(10)

where $\rho _S(x,y)$ is the specular albedo, $\mathbf{H}^{i}=\frac{\mathbf{V}+\varvec{\omega }^{i}}{|\mathbf{V}+\varvec{\omega }^{i}|}=\frac{\mathbf{h}^{i}}{|\mathbf{h}^{i}|} = \big (\frac{h_1^{i}}{|\mathbf{h}^{i}|}, \frac{h_2^{i}}{|\mathbf{h}^{i}|}, \frac{h_3^{i}}{|\mathbf{h}^{i}|}\big )$ and c is a positive constant that measures the shininess of the surface. Next, we will use the following notation: $\tilde{\mathbf{h}}' = (h'_1, h'_2)$ and $\tilde{\mathbf{h}}'' = (h''_1, h''_2)$, in order to compact the writing as already done for the vectors $\tilde{\varvec{\omega }}',\tilde{\varvec{\omega }}''$.

We next present a novel approach regarding the treatment of specularities. Our strategy is based on extending the differential method proposed in [20] to specularities with image ratios. This yields to

(11)

which makes the differential problem similar to (7), that is

$$\begin{aligned} \left\{ \begin{array}{l@{\quad }l} \displaystyle \mathbf{b}_S(x,y)\cdot \varvec{\nabla } u(x,y)=f_S(x,y) &{} \quad \forall (x,y)\in {\varOmega }, \\ \displaystyle u(x,y)=g(x,y) &{} \quad \forall (x,y)\in \partial {\varOmega }, \end{array} \right. \end{aligned}$$

(12)

with the same boundary conditions and defined by the following functions:

$$\begin{aligned} (\mathbf{b}_S, f_S) = |\mathbf{h}''|(S_2(x,y))^\frac{1}{c}{} \mathbf{h}' - |\mathbf{h}'|(S_1(x,y))^\frac{1}{c}{} \mathbf{h}''. \end{aligned}$$

(13)

Note that also for the specular case, the problem is albedo independent.

Given this new set of equations, we can obtain the solution for the normal field by extracting information directly from the specular component. Even if the viewer direction $\mathbf{V}$ has been considered arbitrary, we require $V_3 > 0$ as assumed for the vectors ${\varvec{\omega }^i}$ (with $\omega _3^{i} > 0$ for each i-th vector). From a theoretical point of view, such assumption simply means that also ${\mathbf{H}^i}$ lie in the upper semisphere. In the rigorous mathematical development, we will see how this is required in order to prove the uniqueness of the solution for (12).

We will explain in Sect. 6, devoted to the numerical experiments, how the specular component affects the reconstruction in the presence of noise.

3 The New Differential Approach

In the previous section, we derived a new linear differential problem for specular reflection (12) based on previous work using diffuse reflection (7). With the aim to merge both reflection effects, we combine these linear equations with a weight $\alpha (x,y)\in \{0,1\}$ as follows:

$$\begin{aligned} \left\{ \begin{array}{l@{\quad }l} \displaystyle \mathbf{b}(x,y)\cdot \varvec{\nabla } u(x,y)=f(x,y) &{} \quad \forall (x,y)\in {\varOmega }, \\ \displaystyle u(x,y)=g(x,y) &{} \quad \forall (x,y)\in \partial {\varOmega }, \end{array} \right. \end{aligned}$$

(14)

where

$$\begin{aligned} \mathbf{b}(x,y)=\alpha (x,y)\mathbf{b}_D(x,y)+(1-\alpha (x,y))\mathbf{b}_S(x,y) \end{aligned}$$

(15)

and

$$\begin{aligned} f(x,y)=\alpha (x,y)f_D(x,y)+(1-\alpha (x,y))f_S(x,y). \end{aligned}$$

(16)

In this way, if $\alpha =1$ we have the differential problem (7) for the Lambertian model. Instead, if $\alpha = 0$ we obtain the specular problem (12). We consider $\alpha$ as a given coefficient, provided by the separation procedure between specular and diffuse components. The well-posedness of problem (14) is guaranteed by proving that both diffuse (7) and specular (12) problems are well-posed. Since (7) has been already proven to be well-posed in [20], we only need to focus on (12).

3.1 Well-Posedness of the Specular Model

In order to verify that the problem (12) is well-posed, we start by proving that the vector field $\mathbf{b}_S$ never vanishes in $\overline{{\varOmega }}$.

Lemma 1

If there are no points $(x,y)\in \overline{{\varOmega }}$ of black shadows for the image functions (i.e., $I_1(x,y)\ne 0$ and $I_2(x,y)\ne 0$), we have that $|\mathbf{b}_S(x,y)|\ne 0$.

Proof

Let us prove this result by contradiction. Suppose that there exists a point $(\bar{x},\bar{y})\in \overline{{\varOmega }}$ such that

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle |\mathbf{h}''|(S_2(\overline{x},\overline{y}))^\frac{1}{c}h'_1-|\mathbf{h}'|(S_1(\overline{x},\overline{y}))^\frac{1}{c}h''_1=0, \\ \displaystyle |\mathbf{h}''|(S_2(\overline{x},\overline{y}))^\frac{1}{c}h'_2-|\mathbf{h}'|(S_1(\overline{x},\overline{y}))^\frac{1}{c}h''_2=0. \end{array} \right. \end{aligned}$$

(17)

Since we want to consider the dependence of the image functions $I_1(x,y)$ and $I_2(x,y)$ on all the other model coefficients, we make these functions explicit by using the Eq. (10), obtaining the following non-linear system:

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle |\mathbf{h}''|\rho _S(x,y)^{\frac{1}{c}}\frac{\mathbf{n}(x,y)}{|\mathbf{n}(x,y)|}\cdot \frac{\mathbf{h}''}{|\mathbf{h}''|}h'_1 \\ \displaystyle \,-|\mathbf{h}'|\rho _S(x,y)^{\frac{1}{c}}\frac{\mathbf{n}(x,y)}{|\mathbf{n}(x,y)|}\cdot \frac{\mathbf{h}'}{|\mathbf{h}'|}h''_1=0, \\ \displaystyle |\mathbf{h}''|\rho _S(x,y)^{\frac{1}{c}}\frac{\mathbf{n}(x,y)}{|\mathbf{n}(x,y)|}\cdot \frac{\mathbf{h}''}{|\mathbf{h}''|}h'_2 \\ \displaystyle \,-|\mathbf{h}'|\rho _S(x,y)^{\frac{1}{c}}\frac{\mathbf{n}(x,y)}{|\mathbf{n}(x,y)|}\cdot \frac{\mathbf{h}'}{|\mathbf{h}'|}h''_2=0, \end{array} \right. \end{aligned}$$

(18)

that is

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle \mathbf{n}(x,y) \cdot \mathbf{h}'' h'_1 - \mathbf{n}(x,y) \cdot \mathbf{h}' h''_1=0, \\ \displaystyle \mathbf{n}(x,y) \cdot \mathbf{h}'' h'_2 - \mathbf{n}(x,y) \cdot \mathbf{h}' h''_2=0. \end{array} \right. \end{aligned}$$

(19)

Now, we compute $\displaystyle \frac{\partial u}{\partial x}$ and $\displaystyle \frac{\partial u}{\partial y}$. We omit the dependence on (x, y) in order to ease the notation. By considering $\displaystyle \mathbf{n} = \big (- \frac{\partial u}{\partial x},- \frac{\partial u}{\partial y},1\big )$, we solve the following system:

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}'' + h_3'' )h_1'- (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}' + h_3' )h_1''=0, \\ \displaystyle (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}'' + h_3'' )h_2' - (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}' + h_3' )h_2''=0, \end{array} \right. \end{aligned}$$

(20)

that can be rewritten as follows:

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle \big ((- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}'' + h_3'' ),- (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}' + h_3' )\big )\cdot (h_1',h_1'')=0, \\ \displaystyle \big ((- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}'' + h_3'' ),- (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}' + h_3' )\big )\cdot (h_2',h_2'')=0. \end{array} \right. \end{aligned}$$

This means that the vectors $(h_1',h_1'')$ and $(h_2',h_2'')$ are orthogonal to $\big ((- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}'' + h_3'' ),- (- \varvec{\nabla } u \cdot \tilde{\mathbf{h}}' + h_3' )\big )$ simultaneously. We can consider two cases:

• 1. The vectors $(h_1',h_1'')$ and $(h_2',h_2'')$ are orthogonal and coincident (that is, $(h_1',h_1'')\equiv (h_2',h_2'')$).

• 2. Both $(h_1',h_1'')$ and $(h_2',h_2'')$ are orthogonal and placed in the opposite direction (that is, $h_1'=-h_2'$ and $h_1''=-h_2''$).

By parametrizing the vectors $\mathbf {h'},\mathbf {h''}$ with spherical coordinates having $\varphi$ and $\theta$ as zenith and azimuth angles, respectively, we can rewrite the previous two cases accordingly. That is, the first case is when $\theta _1,\theta _2\in \{\frac{\pi }{4},\frac{5\pi }{4}\}$, whereas in the second case $\theta _1,\theta _2\in \{\frac{3\pi }{4},\frac{7\pi }{4}\}$.

For both cases, we can consider, instead of the two null components of the vector $\mathbf{b}_S$, only one equation (because they are the same in such cases). Let us consider, for example, the first

$$\begin{aligned} \displaystyle \left( - \varvec{\nabla } u \cdot \tilde{\mathbf{h}}'' + h_3'' \right) h_1'- \left( - \varvec{\nabla } u \cdot \tilde{\mathbf{h}}' + h_3' \right) h_1''=0, \end{aligned}$$

(21)

that is

$$\begin{aligned}&-\frac{\partial u}{\partial x}h_1''h_1'-\frac{\partial u}{\partial y}h_2''h_1'+h_3''h_1'+ \frac{\partial u}{\partial x}h_1'h_1'' +\,\frac{\partial u}{\partial y}h_2'h_1''\\&\quad -\,h_3'h_1'' = 0, \end{aligned}$$

which is

$$\begin{aligned}&- \frac{\partial u}{\partial x}h_1''h_1'-\frac{\partial u}{\partial y}h_1''h_1'+h_3''h_1'+\frac{\partial u}{\partial x}h_1'h_1''+\frac{\partial u}{\partial y}h_1'h_1''\\&\quad -\,h_3'h_1''= 0. \end{aligned}$$

This implies

$$\begin{aligned} \frac{h_3''}{h_3'}=\frac{h_1''}{h_1'}=\frac{h_2''}{h_2'}. \end{aligned}$$

Then, using the spherical coordinates parametrizing the vectors $\mathbf {h'}, \mathbf {h''}$, we get

$$\begin{aligned} \frac{\cos \varphi _2}{\cos \varphi _1}=\frac{\sin \varphi _2\cos \theta _2}{\sin \varphi _1\cos \theta _1}=\frac{\sin \varphi _2\sin \theta _2}{\sin \varphi _1\sin \theta _1}. \end{aligned}$$

(22)

Our goal is to prove that $\theta _1=\theta _2$ in order to obtain the contradiction because in this case we have that $\widetilde{\mathbf {h}}'=\widetilde{\mathbf {h}}''$ (i.e., $\widetilde{\mathbf {\omega }}' + \widetilde{\mathbf {V}} = \widetilde{\mathbf {\omega }}'' + \widetilde{\mathbf {V}} \Leftrightarrow \widetilde{\mathbf {\omega }}' = \widetilde{\mathbf {\omega }}''$ that implies $\mathbf {\omega '} = \mathbf {\omega ''}$) and this it is not the case for the photometric stereo technique.

It is clear that, for both cases, if $\frac{\cos \theta _2}{\cos \theta _1}=+1$, then $\theta _1=\theta _2$, while $\frac{\cos \theta _2}{\cos \theta _1}=-1$ means $\theta _1\ne \theta _2$.

Let us suppose, by contradiction again, that $\theta _1\ne \theta _2$. Then, from (22) we have

$$\begin{aligned} \frac{\cos \varphi _2}{\cos \varphi _1}=-\frac{\sin \varphi _2}{\sin \varphi _1}\Rightarrow \tan \varphi _1=-\tan \varphi _2, \end{aligned}$$

which is not possible because $\varphi _1,\varphi _2\in [0,\frac{\pi }{2}]$, since the vectors $\mathbf {\omega }'$, $\mathbf {\omega }''$, and $\mathbf {V}$ belong to the superior part of the unit sphere, which implies that $\mathbf {h}'$ and $\mathbf {h}''$ will also belong to the same region of the unit sphere. $\square$

With the aim to use the characteristics method, we need to show that the information traveling on such curves crosses the image domain. For Lipschitz continuous surfaces, the image function could present jump discontinuities corresponding to the curves where the surface is not differentiable. Let us call such curves as $\gamma (t)$ and since they could represent an obstacle to the information propagation due to the characteristics method, we need the following result showing that a unique (then weak) solution exists.

Theorem 1

Let $\gamma (t)$ be a regular curve of discontinuity for the functions $\mathbf{b}_S(x,y)$ and $f_S(x,y)$. Let $(\overline{x},\overline{y})\in \gamma (t)$, and let $\mathbf{n}(\overline{x},\overline{y})$ be the outgoing normal with respect to the set ${\varOmega }^+$ located on the right of $\gamma (t)$. Then we have

$$\begin{aligned} \begin{array}{l@{\quad }l} &{}\Bigg [ \lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y}) \\ (x,y)\in {\varOmega }^+ \end{array}}{} \mathbf{b}_S(x,y)\cdot \mathbf{n}(\overline{x},\overline{y})\Bigg ] \\ &{} \quad \cdot \,\Bigg [ \lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y})\\ (x,y)\in {\varOmega }^- \end{array}}{} \mathbf{b}_S(x,y)\cdot \mathbf{n}(\overline{x},\overline{y})\Bigg ]\ge 0. \end{array} \end{aligned}$$

(23)

Proof

We give the guideline of the proof which can be seen as a variation of the equivalent proof in [20]. Let us define the quantities

$$\begin{aligned} \begin{aligned} I_1^+:=&\lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y})\\ (x,y)\in {\varOmega }^+ \end{array}}I_1(x,y), \quad \quad I_1^-:= \lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y})\\ (x,y)\in {\varOmega }^- \end{array}}I_1(x,y),\\ I_2^+:=&\lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y})\\ (x,y)\in {\varOmega }^+ \end{array}}I_2(x,y),\quad \quad I_2^-:= \lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y})\\ (x,y)\in {\varOmega }^- \end{array}}I_2(x,y).\\ \end{aligned} \end{aligned}$$

In order to work with the vector field $\mathbf{b}_S(x,y)$ in the neighborhood of $(\overline{x},\overline{y})$, we consider the relations

$$\begin{aligned} \begin{aligned} \lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y}) \\ (x,y)\in {\varOmega }^+ \end{array}}{} \mathbf{b}_S(x,y)&= (b^+_1,b^+_2) \\&= |\mathbf{h}''|(I^+_2)^\frac{1}{c}\tilde{\mathbf{h}}' - |\mathbf{h}'|(I^+_1)^\frac{1}{c}\tilde{\mathbf{h}}'', \\ \lim _{\begin{array}{c} (x,y)\rightarrow (\overline{x},\overline{y}) \\ (x,y)\in {\varOmega }^- \end{array}}{} \mathbf{b}_S(x,y)&= (b^-_1,b^-_2) \\&= |\mathbf{h}''|(I^-_2)^\frac{1}{c}\tilde{\mathbf{h}}' - |\mathbf{h}'|(I^-_1)^\frac{1}{c}\tilde{\mathbf{h}}''. \end{aligned} \end{aligned}$$

(24)

Denoting by $(n_1,n_2)=(n_1(\overline{x},\overline{y}),n_2(\overline{x},\overline{y}))$ the two coordinates of $\mathbf{n}(\overline{x},\overline{y})$ and replacing (24) in the inequality (23), we obtain

$$\begin{aligned} \begin{aligned} b^+_1b^-_1n_1^2+b^+_2b^-_2n_2^2&+n_1n_2\left( b^+_1b^-_2+b^+_2b^-_1\right) \ge 0, \end{aligned} \end{aligned}$$

which gives, in explicit terms,

$$\begin{aligned}&\left( |\mathbf{h}''|(I^+_2)^\frac{1}{c}h'_1-|\mathbf{h}'|(I^+_1)^\frac{1}{c}h''_1\right) \nonumber \\&\quad \cdot \,\left( |\mathbf{h}''|(I^-_2)^\frac{1}{c}h'_1-|\mathbf{h}'|(I^-_1)^\frac{1}{c}h''_1\right) n_1^2 \nonumber \\&\quad + \left( |\mathbf{h}''|(I^+_2)^\frac{1}{c}h'_2-|\mathbf{h}'|(I^+_1)^\frac{1}{c}h''_2\right) \nonumber \\&\quad \cdot \, \left( |\mathbf{h}''|(I^-_2)^\frac{1}{c}h'_2-|\mathbf{h}'|(I^-_1)^\frac{1}{c}h''_2\right) n_2^2 \nonumber \\&\quad +\,\Big [\left( |\mathbf{h}''|(I^+_2)^\frac{1}{c}h'_1-|\mathbf{h}'|(I^+_1)^\frac{1}{c}h''_1\right) \nonumber \\&\quad \cdot \, \left( |\mathbf{h}''|(I^-_2)^\frac{1}{c}h'_2-|\mathbf{h}'|(I^-_1)^\frac{1}{c}h''_2\right) \nonumber \\&\quad + \left( |\mathbf{h}''|(I^+_2)^\frac{1}{c}h'_2-|\mathbf{h}'|(I^+_1)^\frac{1}{c}h''_2\right) \nonumber \\&\quad \cdot \, \left( |\mathbf{h}''|(I^-_2)^\frac{1}{c}h'_1-|\mathbf{h}'|(I^-_1)^\frac{1}{c}h''_1\right) \Big ]n_1n_2\ge 0. \end{aligned}$$

(25)

For each coefficient that multiplies $n_1^2$, $n_2^2$, and $n_1n_2$, we substitute the Eq. (10) and, after some algebraic simplifications, we get, respectively, the following (26), (27), and (28)

$$\begin{aligned}&i_1^-i_1^+\left( h_1''\right) ^2 - i_1^+i_2^-h_1'h_1'' - i_1^- i_2^+h_1'h_1''+i_2^-i_2^+\left( h_1'\right) ^2 \end{aligned}$$

(26)

$$\begin{aligned}&i_1^-i_1^+\left( h_2''\right) ^2- i_1^+i_2^-h_2'h_2'' -i_1^-i_2^+h_2'h_2''+i_2^- i_2^+ \left( h_2'\right) ^2 \end{aligned}$$

(27)

$$\begin{aligned}&\quad -\,i_1^+i_2^-h_1''h_2'-i_1^-i_2^+h_1''h_2'+2 i_2^-i_2^+h_1'h_2' \nonumber \\&\quad \,+\, 2 i_1^-i_1^+h_1''h_2'' - i_1^+i_2^- h_1'h_2'' - i_1^-i_2^+h_1'h_2'' \end{aligned}$$

(28)

where

$$\begin{aligned} \begin{aligned}&i^+_1:=\bigg (-\frac{\partial u}{\partial x},-\frac{\partial u}{\partial y}\bigg )^+\cdot \left( h_1',h_2'\right) +h_3', \\&i^-_1:=\bigg (-\frac{\partial u}{\partial x},-\frac{\partial u}{\partial y}\bigg )^-\cdot \left( h_1',h_2'\right) +h_3',\\&i^+_2:=\bigg (-\frac{\partial u}{\partial x},-\frac{\partial u}{\partial y}\bigg )^+\cdot \left( h_1'',h_2''\right) +h_3'', \\&i^-_2:=\bigg (-\frac{\partial u}{\partial x},-\frac{\partial u}{\partial y}\bigg )^-\cdot \left( h_1'',h_2''\right) +h_3''. \end{aligned} \end{aligned}$$

(29)

This allows us to write the following equalities

$$\begin{aligned} i^+_1-i^-_1= & {} \varvec{\xi } \cdot (h_1',h_2'), \nonumber \\ i^+_2-i^-_2= & {} \varvec{\xi }\cdot (h_1'',h_2''), \end{aligned}$$

(30)

where $\varvec{\xi } \!=\! \varvec{\nabla } u^- - \varvec{\nabla } u^+$. This reduces the problem to a previously solved one in [20], which allows to end the proof.   $\square$

4 W-PS with No Boundary Condition

In the previous section, we considered the PS problem with only two images assuming knowledge of the boundary condition g(x, y). However, for most real applications, the depth on the boundary is not available. It is therefore important to find a way to solve the PS problem without requiring knowledge of the boundary condition. An interesting way to do that is by using more than two images obtained by using different light sources. However, in this case, an additional constraint on the lighting directions is required: the light sources have to be non-coplanar. This inconvenience has been studied in [22] with respect to the PS with three images.

To solve the PS problem without knowledge of the boundary condition, we consider the following numerical strategy. First, we select a single arbitrarily valued initial seed point within the reconstruction domain. Next, we robustly manipulate the path of the characteristics as in [21].

4.1 Controlling the Characteristic Field

Let us start by considering the PS problem with three images. We can consider a set of unique image pairs and arrive to the following system of linear PDEs:

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle \mathbf{b}^{(1,2)}(x,y)\cdot \varvec{\nabla } u(x,y)=f^{(1,2)}(x,y) \\ \displaystyle \mathbf{b}^{(1,3)}(x,y)\cdot \varvec{\nabla } u(x,y)=f^{(1,3)}(x,y)\\ \displaystyle \mathbf{b}^{(2,3)}(x,y)\cdot \varvec{\nabla } u(x,y)=f^{(2,3)}(x,y), \end{array} \right. \end{aligned}$$

(31)

where $\mathbf{b}^{(h,k)}(x,y)$ and $f^{(h,k)}(x,y)$ are linear combinations using data acquired with the h-th and k-th light with $(h,k)\in \left( {\begin{array}{c}3\\ 2\end{array}}\right)$, that is the set of pairs of integer indices with no repetition, i.e., (1,2), (1,3), and (2,3).

In order to define a numerical strategy we need to manipulate the path along which the information travels. To do that, we can use following theorem:

Theorem 2

Let $\mathbf{b}^{p}(x,y)$ be the vector field of (31) where $p\in \left( {\begin{array}{c}3\\ 2\end{array}}\right)$. Then, $\forall p_1,p_2\in \left( {\begin{array}{c}3\\ 2\end{array}}\right)$ and $\forall (x,y)\in {\varOmega }$ we have:

$$\begin{aligned} \mathbf{b}^{p_1}(x,y)\cdot \mathbf{b}^{p_2}(x,y)\ne \pm \big |\mathbf{b}^{p_1}(x,y)\big |\big |\mathbf{b}^{p_2}(x,y)\big |. \end{aligned}$$

(32)

Proof

In order to not involve too many parameters, let us fix the indices $p_1$ and $p_2$ as (1, 2) and (1, 3), respectively. In order to prove that $\mathbf{b}^{(1,2)}$ and $\mathbf{b}^{(1,3)}$ are never parallel, we consider the contradiction assuming that there exists a point $(\tilde{x},\tilde{y})\in {\varOmega }$ such that

$$\begin{aligned} \mathbf{b}^{(1,2)}(\tilde{x},\tilde{y})\cdot \mathbf{b}^{(1,3)}(\tilde{x},\tilde{y}) = \pm \big |\mathbf{b}^{(1,2)}(\tilde{x},\tilde{y})\big |\big |\mathbf{b}^{(1,3)}(\tilde{x},\tilde{y})\big |. \end{aligned}$$

(33)

For the sake of clarity, we omit the dependence on $(\tilde{x},\tilde{y})$. Now, by squaring both sides of (33), we obtain

$$\begin{aligned}&\left[ b^{(1,2)}_1\cdot b^{(1,3)}_1 + b^{(1,2)}_2\cdot b^{(1,3)}_2\right] ^2 \nonumber \\&\quad = \left[ (b^{(1,2)}_1)^2 + (b^{(1,2)}_2)^2\right] \, \left[ (b^{(1,3)}_1)^2 + (b^{(1,3)}_2)^2 \right] \end{aligned}$$

(34)

that, with simplifications, means that we have just to verify

$$\begin{aligned}&2 \, \left[ b^{(1,2)}_1\cdot b^{(1,3)}_1\right] \, \left[ b^{(1,2)}_2\cdot b^{(1,3)}_2\right] \nonumber \\&\quad = \left[ b^{(1,2)}_1 \, b^{(1,3)}_2\right] ^2 + \left[ b^{(1,3)}_1 b^{(1,2)}_2\right] ^2. \end{aligned}$$

(35)

By writing $\mathbf{b}^{(1,2)}$ and $\mathbf{b}^{(1,3)}$ explicitly, we get

(36)

where

$A = D_2 \omega '_1 - D_1 \omega ''_1$,       $B = |\mathbf{h}''| (S_2)^{1/c} h'_1 - |\mathbf{h}'| (S_1)^{1/c} h''_1$

$C = D_2 \omega '_2 - D_1 \omega ''_2$,       $D = |\mathbf{h}''| (S_2)^{1/c} h'_2 - |\mathbf{h}'| (S_1)^{1/c} h''_2$

$E = D_3 \omega '_1 - D_1 \omega '''_1$,       $F = |\mathbf{h}'''| (S_3)^{1/c} h'_1 - |\mathbf{h}'| (S_1)^{1/c} h'''_1$

$G = D_3 \omega '_2 - D_1 \omega '''_2$,       $L = |\mathbf{h}'''| (S_3)^{1/c} h'_2 - |\mathbf{h}'| (S_1)^{1/c} h'''_2$.

After some manipulation on (36), we arrive to the following condition:

$$\begin{aligned}&\alpha (AG - CE) + \alpha (1-\alpha ) [AL + BG - CF - DE] \nonumber \\&\quad +\, (1-\alpha )^2 (BL - DF) = 0. \end{aligned}$$

(37)

Depending on the values of $\alpha$, we can distinguish two cases.

Case 1: Purely diffuse case. This is the case when just the diffuse vector field $\mathbf{b}_D$ is considered ($\alpha = 1$). We have to verify which condition leads to $AG - CE = 0$.

$$\begin{aligned} \begin{aligned}&AG - CE = 0 \\&\Leftrightarrow (D_2 \omega '_1 - D_1 \omega ''_1) \, (D_3 \omega '_2 - D_1 \omega '''_2) \\&\quad - (D_2 \omega '_2 - D_1 \omega ''_2) \, ( D_3 \omega '_1 - D_1 \omega '''_1) = 0 \end{aligned} \end{aligned}$$

(38)

that holds only if the light sources are coplanar (see [23] for details on the proof).

Case 2: Purely specular case. When $\alpha = 0$, just the specular vector field $\mathbf{b}_S$ is considered. We have to verify the condition for $BL - DF = 0$.

$$\begin{aligned}&BL - DF = 0 \nonumber \\&\quad \Leftrightarrow \left[ |\mathbf{h}''| (S_2)^{1/c} h'_1 - |\mathbf{h}'| (S_1)^{1/c} h''_1\right] \nonumber \\&\quad \cdot \left[ |\mathbf{h}'''| (S_3)^{1/c} h'_2 - |\mathbf{h}'| (S_1)^{1/c} h'''_2\right] \nonumber \\&\qquad -\, \left[ |\mathbf{h}''| (S_2)^{1/c} h'_2 - |\mathbf{h}'| (S_1)^{1/c} h''_2\right] \nonumber \\&\quad \cdot \left[ |\mathbf{h}'''| (S_3)^{1/c} h'_1 - |\mathbf{h}'| (S_1)^{1/c} h'''_1\right] = 0 \nonumber \\&\quad \Leftrightarrow |\mathbf{h}'|\, |\mathbf{h}''| (S_1)^{1/c} (S_2)^{1/c} \, \left( h'_2 h'''_1 - h'_1 h'''_2\right) \nonumber \\&\qquad + \,|\mathbf{h}'|\, |\mathbf{h}'''| (S_1)^{1/c} (S_3)^{1/c} \, \left( h'_1 h''_2 - h'_2 h''_1\right) \nonumber \\&\qquad +\, |\mathbf{h}'|^2 (S_1)^{1/c} \, \left( h''_1 h'''_2 - h''_2 h'''_1\right) = 0 \nonumber \\&\quad \Leftrightarrow |\mathbf{h}'|\, (S_1)^{1/c} \, \left[ |\mathbf{h}''| (S_2)^{1/c} \left( h'_2 h'''_1 - h'_1 h'''_2\right) \right. \nonumber \\&\qquad + |\mathbf{h}'''| (S_3)^{1/c} \, \left( h'_1 h''_2 - h'_2 h''_1\right) \nonumber \\&\qquad \left. +\, |\mathbf{h}'| (S_1)^{1/c} \, \left( h''_1 h'''_2 - h''_2 h'''_1\right) \right] = 0. \end{aligned}$$

(39)

Recalling the definition of $S_i$ from (10) and the definition of the normal $\mathbf{N}(x,y)$ from (1), we can rewrite (39) as

$$\begin{aligned}&\frac{|\mathbf{h}'| \rho ^{1/c}_S (S_1)^{1/c}}{\sqrt{1 + |\varvec{\nabla } u|^2}} \, \left[ \left( -h''_1u_x - h''_2 u_y + h''_3 \right) \, \left( h'_2 h'''_1 - h'_1 h'''_2\right) \right. \nonumber \\&\quad + \left( -h'''_1u_x - h'''_2 u_y + h'''_3 \right) \, \left( h'_1 h''_2 - h'_2 h''_1\right) \nonumber \\&\quad \left. + \left( -h'_1u_x - h'_2 u_y + h'_3 \right) \, \left( h''_1 h'''_2 - h''_2 h'''_1\right) \right] = 0. \end{aligned}$$

(40)

After some algebraic manipulation, we get

$$\begin{aligned}&\frac{|\mathbf{h}'| \rho ^{1/c}_S (S_1)^{1/c}}{\sqrt{1 + |\varvec{\nabla } u|^2}} \, \left[ h'_1 h''_3 h'''_2 - h'_2 h''_3 h'''_1 + h'_2 h''_1 h'''_3 \right. \nonumber \\&\quad \left. - h'_1 h''_2 h'''_3 + h'_3 h''_2 h'''_1 - h'_3 h''_1 h'''_2 \right] = 0. \end{aligned}$$

(41)

Assuming a non-shadowed point for the first image (i.e., $S_1 > 0$), we have that (41) is satisfied only if the three vectors $\mathbf{h}',\mathbf{h}'',\mathbf{h}'''$ are coplanar. This is equivalent to having coplanar light sources, since from (41), one can write

$$\begin{aligned} \det \left( \begin{array}{c@{\quad }c@{\quad }c} h'_1 &{} h'_2 &{} h'_3 \\ h''_1 &{} h''_2 &{} h''_3 \\ h'''_1 &{} h'''_2 &{} h'''_3 \end{array} \right) = 0 \end{aligned}$$

which is verified when the light directions are coplanar. This is in contradiction with the photometric stereo assumption and proves (32). $\square$

This theorem states that by considering a linear combination of PDEs as in (14), obtained by coupling different pairs of images, characteristic strip expansion can be performed according to the most convenient direction as in [21].

Fig. 3

Test 1: Images of a paraboloid without noise before (first line) and after the reflection separation. In the last column on the top, the original 3D shape. In the second and third row are shown the color-coded normal maps of the reconstructed shape using only the diffuse and the specular component, respectively (Color figure online)

4.2 Upwind Scheme

In this section, we describe the numerical methods that we employ in order to verify the validity of the proposed model. We use these methods to approximate our problem (14). The schemes considered originate from [20] where finite difference upwind schemes and semi-Lagrangian schemes are used for the forward and backward approximation of the differential problem (7). The difference with respect to the schemes presented in [20] consists in faster implementations which allow to speed up the convergence of the numerical schemes described below. The faster implementations use the Fast Sweeping technique [29, 40] which exploits the regularity of the diffuse vector field $\mathbf{b}$, similar to that presented in [20] for the case of only two images.

Let us start with a square domain ${\varOmega }$ like the set $[a,b]^2$ (in particular considering in the numerical tests $[-1,1]^2$) and with a uniform discretization space step ${\varDelta }=(b-a)/m$ where m is the number of intervals in which the side of the square is divided (that is $x_i=-1+i{\varDelta }_x$, $y_j=-1+j{\varDelta }_y$ with $i,j=0,\ldots ,m$). We will denote by $\overline{{\varOmega }}_d$ all the points of the lattice belonging to $\overline{{\varOmega }}$, by ${\varOmega }_d$ all the internal points and by $\partial {\varOmega }_d$ all the boundary points.

Fig. 4

Test 1: Images of a paraboloid with 1 % of Gaussian noise before (first row) and after the reflection separation (second and third row). The last column shows the ground truth on top, the color-coded normal map computed by using the diffuse (second row) and the specular (third row) data having, respectively, 2.032$^{\circ }$ and 147.72$^{\circ }$ as maximum angular error with respect to the real normal field

Fig. 5

Test 2: Images of a painted ceramic cup before (first row) and after the reflection separation (second and third row)

Fig. 6

Test 3: Images of a painted ceramic statue before (first row) and after the reflection separation (second and third row)

In order to simplify the notation, in what follows we shall denote $\mathbf{b}(x_i,y_j)$ by $\mathbf{b}_{i,j} = [b_{i,j}^1,b_{i,j}^2]$ and $f(x_i,y_j)$ by $f_{i,j}$.

Let us consider the following implicit upwind scheme, obtained by adding vanishing viscosity:

$$\begin{aligned}&b^1_{i,j}\frac{U_{i+1,j}-U_{i-1,j}}{2{\varDelta }_x}+b^2_{i,j}\frac{U_{i,j+1}-U_{i,j-1}}{2{\varDelta }_y} \nonumber \\&\quad = |b^1_{i,j}|\frac{{\varDelta }_x}{2}\frac{U_{i+1,j}-2U_{i,j}+U_{i-1,j}}{{\varDelta }_x^2} \nonumber \\&\qquad \,+\,|b^2_{i,j}|\frac{{\varDelta }_y}{2}\frac{U_{i,j+1}-2U_{i,j}+U_{i,j-1}}{{\varDelta }_y^2}+f_{i,j}, \end{aligned}$$

(42)

for $i,j=1,\ldots ,m-1$. The artificial diffusion introduced in the right-hand side of (42) allows to follow the vector field $\mathbf{b}$ by considering the most appropriate discretization for the first derivative in order to track the characteristic lines [30, 35]. In particular, it consists of a numerical scheme of consistency order equal to one with respect to both partial derivatives.

By writing (42) as

$$\begin{aligned}&U_{i+1,j}\bigg (\frac{b^1_{i,j}-|b^1_{i,j}|}{2{\varDelta }_x}\bigg )-U_{i-1,j}\bigg (\frac{b^1_{i,j}+|b^1_{i,j}|}{2{\varDelta }_x}\bigg ) \nonumber \\&\quad +U_{i,j}\bigg (\frac{|b^1_{i,j}|}{{\varDelta }_x}+\frac{|b^2_{i,j}|}{{\varDelta }_y}\bigg ) + U_{i,j+1}\bigg (\frac{b^2_{i,j}-|b^2_{i,j}|}{2{\varDelta }_y}\bigg ) \nonumber \\&\quad -U_{i,j-1}\bigg (\frac{b^2_{i,j}+|b^2_{i,j}|}{2{\varDelta }_y}\bigg )=f_{i,j}, \end{aligned}$$

(43)

and by assuming ${\varDelta }_x={\varDelta }_y={\varDelta }$, we get the following fixed point iterative scheme

$$\begin{aligned} U^{(k+1)}_{i,j}= & {} \frac{-U^{(k)}_{i+1,j}\left( b^1_{i,j}-|b^1_{i,j}|\right) +U^{(k)}_{i-1,j}\left( b^1_{i,j}+|b^1_{i,j}|\right) }{2\left( |b^1_{i,j}|+|b^2_{i,j}|\right) } \nonumber \\&+ \frac{-U^{(k)}_{i,j+1}\left( b^2_{i,j}-|b^2_{i,j}|\right) +U^{(k)}_{i,j-1}\left( b^2_{i,j}+|b^2_{i,j}|\right) +2{\varDelta } f_{i,j}}{2\left( |b^1_{i,j}|+|b^2_{i,j}|\right) }\nonumber \\ \end{aligned}$$

(44)

rewritten as follows:

$$\begin{aligned} U^{(k+1)}_{i,j}=\frac{|b^1_{i,j}|U^{(k)}_{i-{{\mathrm{sgn}}}{(b^1_{i,j})},j}+|b^2_{i,j}|U^{(k)}_{i,j-{{\mathrm{sgn}}}{(b^2_{i,j})}}+{\varDelta } f_{i,j}}{|b^1_{i,j}|+|b^2_{i,j}|}. \end{aligned}$$

(45)

5 Shape Reconstruction Using Diffuse and Specular Components

The shape recovery approach we present here is based on finding a solution to Eqs. (7) and (12). Such a solution can be obtained by considering the geometrical properties of these equations, expressed as the general linear PDE:

$$\begin{aligned} (\mathbf{b}(x,y),f(x,y))\cdot (-\varvec{\nabla } u(x,y),1) = 0. \end{aligned}$$

(46)

Notably, the three-dimensional vector field $\mathbf {v} = (\mathbf {b},f)$ has to be orthogonal to the normal vector parametrized as in Eq. (1), i.e., tangent to the surface itself. In this way, the computation of the normal field can be derived from such orthogonality. In particular, let us assume to have three images $I_1$, $I_2$, and $I_3$ taken from different and non-coplanar light sources. We can use two out of three available vector fields $\mathbf {v}^{(1,2)}$, $\mathbf {v}^{(1,3)}$, and $\mathbf {v}^{(2,3)}$ to compute the normal vectors to the surface as follows:

$$\begin{aligned} \mathbf {n}^\pm (x,y) = \mathbf {v}^{(1,2)}(x,y)\times \mathbf {v}^{(1,3)}(x,y), \end{aligned}$$

(47)

Fig. 7

Test 4: Images of a plastic ball before (first row) and after the reflection separation (second and third row)

where the ambiguity is eliminated by choosing the normal oriented along the third component with the positive sign

$$\begin{aligned} \mathbf {n}(x,y) = {{\mathrm{sgn}}}\left( n_3^{\pm }(x,y)\right) \mathbf {n}^\pm (x,y). \end{aligned}$$

(48)

In the case where the three available diffuse or specular components are known only locally, we compute the normals considering Eqs. (47) and (48) pointwise. Then, given the normal field, the 3D surface can be approximated by an integration procedure as in [11].

Section 2 illustrated how it is theoretically possible to reconstruct the shape of an object in two independent ways: by using purely diffuse components (8) or by using purely specular ones (13). In practice, since the signal-to-noise ratio is typically very low for the specular components, it is difficult to carry out the correct shape recovery from purely specular light. However, specular images can still be used to reconstruct the shape of an object in the areas where specularities are dominant.

Our aim is to provide a general PS method that results in optimal shape reconstruction whether specularities are present or not (very often, traditional PS relies on images which have favorable lighting conditions in order to avoid pixels under shadows or saturations). For this reason, we define two different cases for which we provide a specific solution at a given pixel (x, y). They include the main 2 situations that one encounters in real experiments. In both cases, separation into diffuse and specular components is assumed.

• A. At least two diffuse image pixels at (x, y) are available as represented by Fig. 5. In such a case we can solve for the diffuse equations and treat for the third image pixel entry as a missing data exploiting the strategy suggested in [23]. Specularities can be ignored.

• B. Less than two diffuse image pixels at (x, y) are available. That is, specularities are overlapping or partially overlapping throughout the different images (see Fig. 9). In this case, we use the specular image $S_i$ equations in order to perform the 3D reconstruction in the areas where specularities are dominant.

6 Numerical Tests

In order to show the performance of our approach, we consider the synthetic case for quantitative results, while qualitative evaluation is provided for real tests. Finally, a numerical test on synthetic images via radial expansion of the characteristics method is shown.

Fig. 8

Reconstructions of Tests 2, 3, and 4 corresponding to case A. On the first column, traditional PS approach with global reflection assumed purely diffuse. The second column shows the reconstruction from the diffuse component where the normals of the surface in the highlighted sets have been recovered by inverting the Blinn–Phong shading model. The last column presents the reconstruction obtained by using our method showing visible improvements in the 3D reconstruction and normal map

Fig. 9

Test 5: In the first line a human face before the separation of diffuse and specular reflections which are shown in the second and third row, respectively

6.1 Synthetic Case

This experimental section explains how diffuse and specular components have different behaviors when subject to noise. To do that, we start by considering the data (Test 1) in Fig. 3 without noise. These images depict the paraboloid shown on the top right corner in Fig. 3.

We then compute the 3D reconstruction by using the methods described in the previous sections in which diffuse and specular images are considered as separated sources of information. The corresponding normal map is presented in the last column of Fig. 3.

We repeat the same procedure but, in this case, we add 1 % of Gaussian noise to all the images in Fig. 3, as shown in Fig. 4. The reconstructed 3D shape, represented as a color-coded normal map for both specular and diffuse components, is shown in the right column of Fig. 4. In the case of reconstruction using specularities, signal-to-noise ratio is a very critical parameter. This is due to the sparsity of the specular images. More in details, since the information provided by the specular component concentrates in white peaks where the normals bisect the viewing and the light directions, specular images are mostly composed by dark regions. In fact, reduced number of pixels belonging to those regions usually do not provide enough information to reconstruct the shape. From a numerical point of view, the specular reflection model requires the computation of $S_i(x,y)^\frac{1}{c}$ in (13) which is highly sensitive to noise. Consequently, shape reconstruction from the specular component is greatly deteriorated even for a very small amount of noise. In contrast, the reconstructed shape obtained using the diffuse component is barely affected by such low level of noise.

Fig. 10

Reconstructions of Test 5 corresponding to case B. The column on the left shows the reconstruction performed by the traditional PS approach considering global reflection to be purely diffuse. In the center, the reconstruction using traditional PS with diffuse and specular separation. On the right column, the reconstruction considering our method where the fast-marching procedure based on the Blinn–Phong shading model has been adopted in the sets where highlights are present

Fig. 11

Test 6: The first row shows the three images used to reconstruct the bunny with our method. On the left second row, plot of the color-coded normal error for the reconstructed surface using the algorithm in [16] and ten images. On the right second row, the normal error when using our approach with only the three images (Color figure online)

6.2 Real Cases

To demonstrate the feasibility of our approach, we analyze real cases by using three white light bright LEDs synchronized by an Arduino-Nano micro controller together with a Basler camera taking images of size 1278 pixels by 958 pixels. The code has been implemented in MATLAB using a 2.3 GHz Intel Core i7 processor with 4GB RAM.

After image acquisition, we separate diffuse and specular components by using the procedure in [32], which works as qualitatively good as [18] and easier to tune by choosing just two parameters. It is important to note that pixel saturation should be avoided in order to facilitate image separation into diffuse and specular components. This can be a difficult task, but new detectors with an extended dynamic range are making it easier to perform in practice. In the current experiments, saturation effects were kept at a minimum.

Fig. 12

Test 7: a Original surface; b computed surface

Fig. 13

Test 7: The three input purely specular images used for the 3D reconstruction

In order to evaluate shape recovery for the case A explained in Sect. 5, we consider three different objects: the painted ceramic cup (Test 2) from Fig. 5, the painted ceramic statue (Test 3) displayed in Fig. 6, and the plastic ball from Fig. 7 (Test 4) whose smoothness allows us to qualitatively evaluate the deformed reconstruction we achieve with traditional PS. For each of these objects, specular pixels exist in only one of the set of three images acquired, which indeed corresponds to case A.

Figure 8 shows the results for the normals and 3D reconstructions of Tests 2, 3, and 4 by using three different methods. The left column corresponds to reconstructions obtained with the traditional PS approach. In this case, global reflection is considered to be purely diffuse. As expected, artifacts can be observed around specularities. The central column shows reconstructions using traditional PS in which the images have been separated into diffuse and specular components. In particular, the diffuse component has been used to reconstruct most of the shape. The empty specular sets have been filled by computing the almost planar surface where the outgoing normal vectors have been approximated by inverting the Blinn–Phong shading model at each highlighted pixel. Artifacts are still present in all Tests, but an improvement on the reconstructed surface is observed, especially evident in Test 2. The right column in Fig. 8 corresponds to surfaces reconstructed using our approach. In this case, we use only the diffuse component, or in other words, we solve Eq. (7). This is possible since, for each set of three images, one can always find two pixels with known boundary conditions and that do not present specularities. Notably, this strategy results in a significant reduction in artifacts when compared to the other tested approaches.

Table 1 Test 7: $L^\infty$ error for different sizes of the images

Fig. 14

Test 7: Error map varying the size of the input images starting from $256\times 256$ (first row on the left) to $2048\times 2048$ (second row on the right)

In order to analyze the performance of our approach for case B (as in Sect. 5), we consider a human face (Test 5, Fig. 9). Interestingly, the tip of the nose generally collects the specular component independent of the orientation of the illumination source. In this way, specularities typically overlap for the three images acquired with different illuminations (definition of case B). As in the previous set of experiments, image reconstruction of the human face is performed using three different approaches (Fig. 10). The traditional PS approach considering the entire captured image as purely diffuse (no reflection components separation), leads to severe artifacts in the prominent specular areas, i.e., the nose tip as well as the bottom lip. When using the traditional PS approach with the image separated into diffuse and specular components (center column), the quality of the reconstruction improves but artifacts are still clearly visible. The third column shows the reconstructed face obtained with our approach. In this case, we solve Eq. (14) with $\alpha =1$ (purely diffuse reflection) everywhere except in the areas where specularities are dominant, in which we use $\alpha =0$. As it occurred with case A, the face reconstructed by using our method presents little artifacts, with a substantial improvement over the traditional PS strategies.

Finally, we compare our method with a recent approach aimed at shape reconstruction with specular highlights. In particular, we select the method of Ikehata et al. [16] that reconstructs the shape of the object by optimizing a bilinear Lambertian model where specular highlights are considered as sparse noise.

For comparison purposes we use the synthetic Bunny shape (Test 6) shown in Fig. 11. In this case the image separation needed by our method does not suffer from saturation problems, since images have been computed synthetically. Figure 11 shows the normal error of the reconstruction for the Ikehata method using ten images and our method just using three images. Notably, the lower error of our method for a lower number of images proves the feasibility of our approach for shape reconstruction with a minimal set of only three images.

6.3 Synthetic Case via Radial Expansion of the Characteristics Method

With this final experiment (Test 7), we want to show the performance of the radial expansion of characteristics applied to the surface visible in Fig. 12a.

The input images used for the reconstruction and visible in Fig. 13 have been generated by using $\alpha =0$ and the specular shininess power $c=15$, without adding noise.

In this test, the fast-marching algorithm (45) has been implemented with the aim to expand the characteristics field from a single point as explained in [21], meaning that the need of boundary conditions has been restricted to the knowledge of a single arbitrary point that we consider as the central one.

Table 1 shows that convergence is obtained in only one iteration (our code compute a second iteration only for check). By increasing the size of the three input images, we can note that the errors computed with the $L^\infty$ norm decrease and the CPU time reported in the last column of the same table remains small (just 34.77 s for image size $2048\times 2048$). This shows the computational efficiency and the high performance of our method.

In Fig. 14 we can see the error map obtained by doubling the size of the input images starting from $256\times 256$ to $2048\times 2048$ pixels.

7 Conclusions and Future Works

This paper presents a new approach for three light photometric stereo aimed at reconstructing surfaces with general BRDF. We derive a new mathematical model for specular surfaces based on the Blinn–Phong reflectance model leading to PDEs having the same linear structure of the one used in [23]. The new model for specular surfaces enables one to extract information of tridimensional shapes from specularities. Moreover, due to the use of the image ratio formulation, the method can deal with materials showing variable albedo. Even if an initial separation of diffuse and specular components is still required, the results show improved 3D reconstruction in synthetic and real experiments. Our method based on a PDEs optimization framework able to merge diffuse and specular components is a promising approach for real-time 3D shape recovery.

Future works will attempt to develop a more accurate technique for reflection separation and to include more robust approaches to merge diffuse and specular components into the PDEs optimization framework. In addition, we will try to model the ambient component in the global image irradiance equation. Due to very different and non-linear physical effects involved in this component, a substantial effort for modeling light propagation is required. The ambient component would provide a step forward allowing the PS to work outside the laboratory.