1 Introduction

Relief is a type of sculptured artwork that is carved into a plane/surface or created by removing the unwanted pieces of material [1]. There are three types of relief [2]: high relief, bas relief and sunken relief. Sunken reliefs depend largely upon the shape which do not protrude from the surface, but are carved into it. In ancient Egyptian times, it was commonly used to illustrate stories and show the greatness of the pharaohs (as shown in Fig. 1). It is also used for inscriptions and engraved gemstones. Nowadays, sunken reliefs are widely used to decorate the surfaces of furniture, walls, buildings, and jewelery.

Fig. 1
figure 1

Sunken relief of ancient Egypt

Full size image

To produce a sunken relief, the sculptor carves the relief within a deeply incised line. As a result, the contours are sunk below the surrounding surface while leaving the highest parts on the surface level [2]. Figure 1 shows that the feature lines are important for depicting the edges of the objects; such lines represent most visual features of a sculpture. Because lines are flexible and concise, they are considered the best way to represent general and abstract figures. In addition, in order to generate a realistic sunken relief, the smooth height transition between edges is also essential. Both the lines and the height transition are combined together to emphasize the presence of a 3D object.

Lines play an important role in forming a sunken relief. The external contours are defined as the principal lines, which are determined by the shape of the object itself and the viewer’s perception. In addition, there are several other linear features which may be present in a relief. These are usually referred to as the interior lines because they are all contained within the overall shapes bounded by the external contours. Edges formed due to the connection of planes/surfaces at an angle may be rendered sharply so that they become a linear feature. These internal feature lines are defined as the secondary lines in a real sunken relief. Last but not the least, there are lines which are not derived from the three-dimensional form of the relief, but are overlaid to highlight the primarily graphic features [2].

Most existing research on relief generation is concerned with creating a smooth relief surface from either image or geometry input. Such a surface has a shallow depth but can deceive the human perception by creating a fake 3D shape with a much greater depth ratio. Few works have paid attention to the importance of lines. Our work introduces a novel framework that uniquely includes feature lines into relief creation, which distinguishes our work from its predecessors.

In our framework, we start from a full scale geometric mesh of a 3D object. In consequence, the user can benefit from abundant 3D models available either online or from laser scanning. It is more effective to operate in the image space than on the geometric mesh. Therefore, we transform the information conveyed by the mesh into three different images as three input layers in our framework: a line drawing image which provides input for contour lines; a rendered Lambertian image which shares the same light direction of the relief and provides visual cues; and a depth image which conveys the height information. By matching the target relief surface with the three given input images by an optimization process, we are able to maximize the presence of features in a sunken relief.

Our work has the following contributions:

  • a novel framework to combine multiple channels of inputs for relief generation;

  • incorporation of lines into smooth relief surface carving;

  • combination of visual cues and real geometric information into the created relief.

Previous works use a single input only. They focus on either the visual cues by transforming an image into a relief or the geometric shape by compressing the geometry into a plane. Using multiple input channels, our method offers more detail and features consequently permitting further art stylization.

The paper is structured as follows: Sect. 2 reviews the relief generation techniques and line drawings from 3D models. Section 3 outlines the structure of our algorithm. Section 4 discusses how to prepare inputs from a 3D object for sunken relief generation. Section 5 derives the mathematical formulation for height optimization from multiple inputs. Section 6 demonstrates some results and Sect. 7 concludes the paper.

2 Related work

2.1 Digital relief generation techniques

Digital relief generation techniques are generally classified as reliefs from images, direct 3D sculpture systems and shape transformation [3]. Digital image processing can generate a relief style image via direct operations on pixels. However, the results of image processing give erroneous three-dimensional information that does not actually convey real height data.

In 2001, Perry and Frisken [4] developed a digital character sculpting system named Kizamu. It is mainly based on adaptively sampling distance fields (ADFs). It also incorporates a blend of new algorithms representing significant technical advances in modeling, introducing novel user interactions. Sourin [5] developed methods using programmed functions to represent shapes and manipulations. These function-based digital sculpting systems can produce carved-in sunken reliefs. Pasko et al. [6] also proposed a method to carve reliefs procedurally using the function representation of geometric objects. These digital systems create reliefs from scratch, which make them more comparable to the geometric modeling tools.

Manuel et al. [7] extended texture mapping techniques to fake the representation of 3D surface details visually, which on the contrary did not alter the geometric information at all. Other techniques, like bump mapping and displacement mapping can also create similar effects.

It is laborious and time-consuming to generate reliefs using 3D modeling software and sculpting system. To improve the efficiency, a few methods have been developed to generate reliefs that convert existing 3D models into relief models. In 1997, Cignoni et al. [8] developed a computer-assisted relief modeling system for generating 3D bas and high reliefs from 3D surface models. Later on, Belhumeur et al. [9] investigated the ambiguity of bas-relief generation with respect to surface reconstruction. It is reported that a linear compression of the 3D geometry can produce relief ambiguity but such implementation often suffers from loss of subtle details when the geometry is complicated. Research in [10, 11] addressed the problem as geometry processing compared to the HDR (High dynamic range imaging [12]) or tone-mapping image compression. Weyrich et al. [13] introduced a perceptually motivated compression algorithm for relief generation. Later, Kerber et al. [14] improved their previous bas-relief generation work by adopting a bilateral filter to achieve good compression ratios without compromising the quality of surface details. This work was further enhanced to support artists and enthusiasts in creating bas-relief artworks from virtual scenes [15]. Sun et al. [16] also developed an automatic system to generate bas reliefs based on adaptive histogram equalization (AHE). All the aforementioned methods in this paragraph are related to flattening the models into a scaled height field with limited depth.

2.2 Shape-from-shading

Shape-from-shading, an approach to reconstructing the original 3D shape from a single image has been developed [17–19]. Shape-from-shading may be used for producing high reliefs but it requires a lot of user intervention. Wu et al. [20], using an altered shape-from-shading method, presented an interactive system for reconstructing surface normal from a single image. Based on this work, Wang et al. [21] developed a method to form a bas-relief surface from an image input.

Alexa and Matusik [22] proposed a method to generate relief surfaces automatically that could produce bas various images under different directional illumination. Although their method had an intrinsic link to shape from shading [17, 18]; the introduced discrete model with necessary degrees of freedom could overcome the theoretical limitations in shape from shading. The method in [22] can take multiple images as input as we do in this paper, however it focused on the illumination conditions while our approach is designed for including contour lines with other visual cues into a sunken relief.

2.3 Lines in sunken relief and 3D line drawings

Line drawings refer to extracting lines (the most obvious features of the model) from 3D models. The extracted lines could be applied to artistic stylization and abstraction work with applications ranging from illustrations to cartoons and gaming scenes.

Our work is inspired by the line drawings from 3D models since sunken reliefs depend considerably upon the carved lines in conveying the surface shape. Lines can convey other different things, including various combinations of lighting, materials, surface markings, and discontinuities [23].

The concept of contours and its connection to human perception was explained in [24]. Such contours in drawings are analogous to the principle lines in sunken reliefs. Later on, creases were used, which was regarded as a big improvement for line drawings from 3D objects [25–27]. Creases are sharp features defined on the mesh surface. Suggestive contours [28–30], which are analogous to the interior lines in sunken reliefs, are minor on-surface features similar to contours that are view dependent and help to denote the local shape. Other drawing elements, such as hatching, are prevalent in more sophisticated line drawings. Many small lines can be combined in such a way that they simultaneously stylize the tone and material [31, 32]. For a more detailed review on line drawings, we refer to [23].

Algorithms for 3D line drawings can extract exactly the lines we need in generating the edges of a sunken relief. However, transforming them into a 3D carving in conjunction with smooth height transition is an unsolved problem. Current relief related research pays little attention to sunken relief generation, and it focuses on the recovery or creation of the height information. Our work, on the other hand, combines the line drawings input with height generation that visually highlights the shape feature on the generated sunken reliefs. To our knowledge, this is the first attempt to implement a method designed for sunken relief generation.

3 Outline of method

Our work aims to generate sunken reliefs from a known 3D object. Most important to note, we will overlay feature lines on the generated reliefs, thereby further enhancing the visual information conveyed to the viewer.

A 3D surface carries more information than a single image does [22]. In our implementation, we start from a triangular mesh by translating it into multiple inputs: a picture of plain line drawings, an image rendered using Lambertian shading, and a depth image. These inputs are combined together in a composition procedure to produce the final relief art piece. These procedures are illustrated in Fig. 2. The input of line drawings helps to create feature lines in the sunken relief at the desired places. The other two inputs work in a conjugate manner to maintain a smooth relief surface.

Fig. 2
figure 2

Procedures of the proposed method

Full size image

4 Input pre-processing

4.1 Line drawings Input

There exists a large body of literature addressing the problem of extracting line drawings from exiting 3D models [25, 28, 31]. Many works contribute to the generation of different stylization and content related drawing, which is beyond the scope of our discussion. As the lines in the sunken relief highlight the contours, we only need to extract the plain line drawings. Figure 3 shows an example of the line drawings we created from a 3D mesh. Two types of line are included in our line drawings: contours (occluding contours and silhouettes) and suggestive contours. We use an object-space algorithm developed in [28] to extract the line drawings. A brief summary of the algorithm is shown below.

Fig. 3
figure 3

An example of Armadillo model

Full size image

Contours

Given a smooth surface, the contour lines are defined as a set of points that lie on the surface and whose surface normal is perpendicular to the view vector:

$$ \mathbf{n}(\mathbf{p}) \cdot\mathbf{v} = 0$$
(1)

p is a point on the surface, n is its normal and v is the view vector. This set of points form disconnected curves on the surface. By projecting the visible part of the lines onto the image plane, we produce a line drawing defined by the occluding contours and silhouettes.

Suggestive contours

Suggestive contours relate to the ridges and valleys as inflection points from a particular view on the surface, which involve computation of the local extrema of the curvature. A direction vector w is defined as the projection of view vector on the tangent plane at point p.

$$ \mathbf{w} = \frac{\mathbf{v} - (\mathbf{n}(\mathbf{p}) \cdot \mathbf{v}) \cdot \mathbf{n}(\mathbf{p})}{\| \mathbf{v} -(\mathbf{n}(\mathbf{p}) \cdot \mathbf{v}) \cdot \mathbf{n}(\mathbf{p})\|}$$
(2)

It is noted that the local minima of n(p)⋅v in direction w suggests a possible location on a ridge/valley. This local minima corresponds to the zero value of the directional derivative of n(p)⋅v along direction w at point p where the second-order directional derivative along w is positive:

$$ \begin{array}{l}\nabla_{\mathbf{w}}(\mathbf{n}(\mathbf{p}) \cdot\mathbf{v}) = 0,\quad \mbox{and}\\[5pt]\nabla_{\mathbf{w}}\nabla_{\mathbf{w}}(\mathbf{n}(\mathbf{p})\cdot\mathbf{v}) > 0\end{array}$$
(3)

where ∇ w denotes the directional derivative along w. This condition is also equivalent to the zero value of radial curvature κ r along direction w where the directional derivative along w of the radial curvature is positive:

$$ \begin{array}{l}\kappa_{r} = 0,\quad \mbox{and}\\[5pt]\nabla_{\mathbf{w}}\kappa_{r} > 0\end{array}$$
(4)

The suggestive contours are generated by projecting those points on the surface that satisfy the aforementioned condition.

In our practice, we notice that direct usage of the line drawings generated from an arbitrary model may not be practical. Too many strokes make the relief appear bumpy, as some important geometric features are concealed or broken by redundant engraved lines. To overcome this issue, the user is provided with a tool to pre-process the mesh to control the input by filtering out the unwanted details. We choose the Laplacian smooth operator to smooth the mesh input. There are a number of different approximations for the Laplacian operator; each has its special usage. Hahan [33] describes its discrete approximation.

$$ L(x_{i}) = \frac{1}{\sum_{ w_{ij}}}\sum_{j \in N(i)} w_{ij}(x_{j} -x_{i})$$
(5)

where w ij is the weight for edge defined by (x i ,x j ). There are several schemes to define the weights w ij . Among those schemes, cotangent weight: w ij =cot(α i,j )+cot(β i,j ), is widely used, where α i,j and β i,j are the angles opposite to edge (x i ,x j ). An example of the practice of controlling the stroke density is shown in Figs. 3(b) and 3(c).

Once the line drawings is produced, additional erosion and dilation operations that directly act on the pixels can be applied to tidy the image up by removing the noise and increasing the width of the lines making them more conspicuous (see Fig. 4).

Fig. 4
figure 4

Processing the extracted lines

Full size image

4.2 Lambertian shading input

Images rendered with the Lambertian model are used as one input source to help recover the height information in the relief. Images rendered with the Lambertian model implicitly codes the surface normal at each point; it helps to recover the height information as the change of the surface normal provides essential information for the perception by the human eyes. The pixel value I is computed from the dot product of normal n and light direction m which point from the surface point toward the light source:

$$ I = \alpha \bigl(\mathbf{n}(\mathbf{p}) \cdot\mathbf{m}(\mathbf{p})\bigr)$$
(6)

where α is the intensity of the light.

In our practice, we align the light source with the view point, which means we actually render n(p)⋅v. In the production process, the light direction can align with the real setting for the relief display accordingly to allow the albedo of the generated relief in line with the geometry exactly.

4.3 Depth image input

It is beneficial to start from a 3D mesh as it provides the height information. Belhumeur et al. [9] suggested that a linear compression of the height could lead to ambiguity for some particular view points of the generated relief. Instead of using the simple linear compression approach, we use optimization approaches described in Sect. 5 to find a robust solution. Using the height information as input, we simply render its depth on the image projection plan as a gray image.

5 Relief height generation

To generate a relief, we initiate our inputs by translating a 3D object into three separate images: a line drawing, a Lambertian shaded image, and a depth image. Those images are generated from the same view point and on the same projection plane to ensure that they are perfectly aligned with each other in terms of their x, y coordinates in the pixel domain. The image is set to have a size of m by n.

In a classic fashion, the discrete surface of a relief is created from an array of points that align with the pixels in the input image and their height is adjustable accordingly of generating a smooth shaded surface. Alexa and Matusik [22] suggested a new arrangement of using four connected triangles to calculate the value at a given pixel, where a pixel of the input image is modeled as a small pyramid in the relief. In our paper, we selected a staggered layout as shown in Fig. 5, where the actual relief grid for storing height data is selected by shifting the pixel grid in x and y directions for half a pixel size.

Fig. 5
figure 5

The surface discretization of the relief in relation to pixels of an input image, where red circles represent the vertices’ locations right on the corners of pixels

Full size image

We define the pixel value of the relief grid node (the red circle in Fig. 5) at (x,y) by averaging the values of its four pixel neighbors,

(7)

we can define the gradient operation of the intensity at the given grid location (x,y) using the same four pixel values as

(8)

Alternatively, we can define the gradient operation of the height value at the pixel location in a similar compact fashion when we estimate the surface normal in Sect. 5.2. Such compact layout helps to increase the resolution for our reconstruction of the relief surface. In the following, we describe how to derive an independent energy function for each of the aforementioned inputs.

5.1 Depth input

With respect to the input depth image, it is possible to transform it into a piece of relief by a linear compression. Such direct operation may cause loss of local features and the resulting relief could look dull due to the lack of depth incorporated information. We process the height information with a non-linear compression by recovering the surface from computing the Poisson equations.

Let us denote the depth value at position (x,y) as I d (x,y). The second-order derivative value is computed by

(9)

G is a non-linear high dynamic compression function [12], which takes the following form:

(10)

γ and β jointly control the compression ratio. We set γ=0.1 and β=0.9 in our experiments.

We then reconstruct the relief surface by solving the following Poisson equations:

(11)

Here h(x,y) specifies the discrete height distribution. This Poisson equation can be numerically resolved. However, in order to integrate our solution with all three inputs, we define the energy terms and find the solution by minimizing the overall energy.

In this way, we have the energy term that relates to the depth image input as follows:

(12)

5.2 Lambertian image input

Let us denote L as the light source. L=α m(p), where α is the light intensity and m is the light direction at point p. The albedo (or the reflection radiance) at the point is

(13)

with n as the surface normal at the point.

Using the staggered grid layout as shown in Fig. 5 to estimate the reflection radiance I of the relief at a pixel point (x+1/2,y+1/2), we can use the four height values at the pixel’s corners to compute the normal direction first. We can write the estimated normal as

(14)

where

and

$$\hat{n} = \sqrt{1 + \biggl( \frac{\partial h}{\partial x} \biggr)^{2} +\biggl(\frac{\partial h}{\partial y} \biggr)^{2}}$$

as the scale factor to normalize the normal vector. In the latter optimization steps, we need to linearize the system by estimating the scale factor with the results from the previous iteration rather than computing it at the current time. The scale factor is initiated as 1 when computing starts. Given n as the normal we can compute the reflection radiance pixel by pixel using (13).

The input Lambertian image has the intensity value at a given position (x+1/2,y+1/2) as I L (x+1/2,y+1/2). Recalling our staggered grid setting, we note that both the estimated reflection radiance and the intensity image are defined at the pixel grids and their gradients are defined at the relief grid nodes by (8). Using the image gradient compression method [12], we can have the compressed image gradient of the input as G(∇I L ), where the compression function is defined in (10). We expect the gradient of the reflection radiance identical to the compressed input value, that is,

(15)

We therefore define the related energy term to the Lambertian image input as follows:

(16)

5.3 Line drawings input

To generate the engraved line effect for a sunken relief, we include a separate line drawing input layer. We define another energy item specially associated to this input. Given the line drawings, we first initiate a stencil based on this. The stencil is an image whose strokes are one pixel wider than those in the input. We denote I w (x+1/2,y+1/2) as the pixel value at position (x+1/2,y+1/2) of the line drawings input. The relief grid value S(x,y) of the stencil template is defined as follows:

(17)

The stencil value is zero only when all of its four neighboring pixels are zero.

Using stencil S, we can filter out the influence of those pixels not adjacent to a given stroke to avoid unnecessary computation. In order to engrave lines on the relief according to the line drawings input, we expect that the following relationship holds:

(18)

I being the intensity of the reflection radiance calculated by (13) whose values are defined at the pixel grid and whose gradients are defined at the relief grid and G is the non-linear compress function of (10).

Therefore, we have the definition of the energy term for the line drawings input as,

(19)

5.4 Compositing relief by minimizing the energy

To recover the height information, we minimize the superposition of all existing energy terms with respect to the height values at the corners of the pixels. The overall energy is written as follows:

(20)

This illustrates that the weighting coefficients control the influence of each term. The gradient of the reflection radiance non-linearly depends on the height alternation. This dependency is linearized by estimating the scale factor which normalizes the normal vector by the results from previous optimization step.

The optimization problem is defined as,

(21)

The constraints of height ensure the relief is sunk into a given flat plane. h max explicitly specifies the maximum depth that can be carved into the plane. We use the Matlab optimization Toolbox to solve this optimization problem.

6 Results

Our algorithm is capable of providing relief models as triangular meshes. Such meshes can be further simplified for less storage or to lower the computational cost for future operation. Other post processing steps, such as texturing and shading can be added to create further embellished results.

As shown in Fig. 6(a), a plain carved relief only includes the line drawings input. Figures 6(b) and 6(c) represent our composting results with all inputs. They have different depths of carving. It is noted that when combining both lines and the surface together, the relief looks much more convincing.

Fig. 6
figure 6

Comparison of reliefs with different inputs

Full size image

Figure 7 illustrates the results from different line drawings inputs with and without Laplacian smoothing. We can see that redundant strokes of input make the result (Fig. 7(a)) appear bumpy and hide the main shape features at some point, which can be enhanced (see Fig. 7(b)) with a mesh smoothing process before extracting the counter lines when preparing the line drawings input.

Fig. 7
figure 7

Comparison of reliefs with different line density

Full size image

Figure 8 demonstrates the generated bas reliefs with different approaches: (a) by Cignoni et al. [8], (b) Kerber et al. [11], (c) Kerber [14], and (d) Sun et al. [16]. One can see from Fig. 8(a) that the simple linear compression is not sufficient for relief generation as many important features are missing. The others preserve the details well, which could be applied to generate sunken relief by simply shifting the height into its surface without highlight of counter lines. The results created by direct sinking a bas relief would be compromised and degenerated because the important feature lines with carved effects are missing, a factor upon which sunken relief mostly relies. The difference between our method (Fig. 7) and others (Fig. 8) is self-evident. Our approach benefits from using the contour lines, which impresses the presence of 3D features and addresses more variations in the art forms. Such lines are organically integrated with the smooth surfaces.

Fig. 8
figure 8

Bas reliefs produced by the methods of (a) Cignoni et al. [8], (b) Kerber et al. [11], (c) Kerber [14], and (d) Sun et al. [16]

Full size image

Figure 9 shows multiple results we have created with our method using different 3D models. It demonstrates that our method is robust in handling both simple abstraction forms (Fig. 9(a), the Pear), and complex geometric features (Fig. 9(d), the Buste). Very low frequent but smooth features like the muscles along the shoulders of the horse or the neck of the pear could hardly be preserved if only the height information is taken into account. The example of the heptoroid shows that the depth order remains visible and correct.

Fig. 9
figure 9

Sunken relief results

Full size image

In our experiment, different 3D meshes are used as inputs to create the sunken reliefs (Figs. 6, 7, 9). The user is able to control the sunken relief outputs with different settings in preparing their inputs. The PC we used is HP Workstation 4300, with Intel Pentium Dual CPU 3.2 GHz and 2 GB RAM, which is equipped with NVIDIA GeForce 7900 GTX video card. It takes about 6–20 seconds of processing time to create a relief using input images with resolution of 1024×768. For example, the cow model (Fig. 6(c)) was created with 6.67 seconds and Armadillo model (Fig. 7((b)) with 18.07 seconds. The variance of time depends on the resolution as well as the complexity of the reliefs. As we have implemented our algorithm in Matlab, we can foresee a potential improvement of performance when rewriting the code in C++.

7 Conclusion

In this paper, we have proposed a method to generate sunken reliefs using 3D models as input. Our experiments show that the combination of lines and height transition can effectively generate satisfactory results. Presented in the digital form, the generated 3D mesh for the resulting sunken relief can be used as input of design software for further refinement or can be directly used for machining a real piece of relief. Such relief form can also be used as references or prototypes for craftsmen in their actual carving practices.

Two major factors are involved in generating a sunken relief: feature edges and smooth height transition, prepared as separate inputs from a known 3D mesh. In the composition step, we minimize the overall energy of the related items to create the height field for smooth surfaces as well as incorporate the lines into a relief.

Although our method starts with a 3D object, the generation of the relief is based on image-like inputs, implying that our method can be directly adopted for relief generation from images. It is possible to extract feature lines and contours from a 2D image using exiting techniques, which makes our method feasible for sunken relief generation by including the images as additional input along with the original image.

Currently, our work is limited to creating sunken reliefs with certain line types (contours, creases etc.). It would be useful, but also challenging to stylize the relief with texture details to reflect the difference derived from the various carving tools used. Such work will be investigated in our future work.