Keywords

1 Introduction

High resolution pictures of paintings are widely used in cultural heritage field for several purposes, addressed both to the enjoyment of the general public and to professional use [1, 2].

While in the first case common undistorted digital photos are considered more than enough, professional uses are more demanding both in terms of resolution and in terms of dimensional accuracy. Especially in case of painting restoration and monitoring purposes, these two aspects assume a fundamental role. To obtain an image that is at the same time measurable (i.e. a measure taken in the image is equal to the corresponding measure value on the painting) and with a gigapixels resolution, the common workflow is based on the realisation of a master orthophoto, which is then used as primary reference for the successive stitching of multiple pictures. As the tiles of a mosaic, these pictures are undistorted and opportunely placed and stretched in order to fit exactly on the master orthophoto. The resulting image is an extremely defined picture, on which is possible to zoom in the finest detail of the art piece, that match exactly the real painting dimensions. Thus, the accuracy of the reference orthophoto is crucial for the success of the whole procedure. The orthophoto is obtained as a transformation of a undistort digital image by means of metrical references (points or planes), usually obtained by means of perspective-based operation. In particular, the cross-ratio method is one of the most adopted methods to evaluate metrics information from pictures [3, 4]. Unfortunately, artworks to be restored and/or presented in digital museums are painted on canvas which are far to be perfectly planar. Therefore, technical documentation accompanying an artwork to be stored in digital archives or museums can be enriched by information related to the 3D shape of the canvas. In other words, conservators, museum curators and practitioners, considers of relevant importance the availability of 3D data related to an artwork [5,6,7].

Common methods [8, 9] consists on projecting several convergent lines to the painting canvas; considering at least 3 reference points on each line, of which 3D coordinates with respect to a defined reference system are known, is possible to calculate the 3D coordinates of the points given by the intersection of the lines and the canvas. On a practical level, this can be obtained by projecting a laser line tangent to a reference table, on which lines and points are drawn. On a digital image, displaying both painting and table, is then possible to perform the operations described below. In order to have a univocal matching between the image to be transformed and the images on which metrical reference are evaluated, camera position must be unchanged during all the operations.

This procedure is very simple and for this reason is often carried out by hand. Unfortunately, this leads to three main drawbacks: (1) the manual selection of the reference points is subject to errors; (2) it is time consuming and requires the constant presence of an expert user; (3) it is limited to the retrieval of a selection of few points, due to timings.

With the aim of overcoming the above-mentioned drawbacks, the present paper proposes the design of a portable low-cost device to reliably acquire the 3D geometry of a canvas. Moreover, an appositely devised semi-automated procedure to obtain an accurate 3D acquisition is described. In fact, the main idea is to realize a cheap system, able to perform a 3D triangulation when coupled with the pre-calibrated user Single-lens reflex (SLR) camera, allowing the 3D measurement of thousands of points on the painting with high accuracy. At the same time, the camera used for triangulating the 3D scene together with the laser can be used to acquire 2D data for creating the orthophoto.

The remainder of paper is structured as follow: in Sect. 2 procedure and device are briefly introduced, in Sect. 3 procedure and its implementation are further discussed, in Sect. 4 physical prototype is described, in Sect. 5 tests and results are discussed.

2 Overall Description

As briefly mentioned, the proposed device aims at retrieving the 3D position of points on the surface of a painting by means of triangulation-based laser camera. Accordingly, the reciprocal positioning and orientation of laser and camera devices (i.e. the extrinsic calibration) needs to be maintained during the acquisition process. Consequently, laser position and orientation are required to be monitored constantly to avoid undesired reciprocal movements [10].

With the aim of keeping implementation costs low, thus avoiding the implementation of devices such as, for instance, encoders, this information is obtained by means of computer vision algorithms.

In particular, as shown in Fig. 1, a fiducial marker is placed tangential to the laser blade and framed by the camera. Since the marker is attached to be visible by the camera for different laser positions, by exploiting algorithms typical of computer vision and augmented reality, is it possible to retrieve the 3D positioning of the maker w.r.t. the camera system. This allow recovering the orientation of the laser beam, especially in case the fiducial marker is placed on a planar board, tangent to the laser itself.

Fig. 1.
figure 1

ChArUco fiducial marker.

Full size image

By opportunely tuning the camera settings and lighting conditions, both laser projection on the painting canvas and marker are acquired. In this way, by using a single picture, is it possible to triangulate laser orientation and laser projection 3D coordinates.

Gradually changing laser orientation and repeating the same operation at each step, is it possible to obtain a 3D point cloud of points of the painting surface which are scanned by the laser beam. Such a point cloud will be more dense by reducing the displacement between consecutive laser positions.

From a procedural point of view, three main aspects have to be encompassed:

  1. 1.

    Detection and pose estimation of the fiducial marker

  2. 2.

    Laser line detection

  3. 3.

    Triangulation algorithm

For what concerns the designed acquisition device, except from an SLR camera, its lens and its tripod, only a few additional tools are needed:

  1. 1.

    Laser line generator;

  2. 2.

    Support for laser line generator;

  3. 3.

    Planar table with fiducial marker;

  4. 4.

    Tripod + panoramic head.

While laser generator, tripod and panoramic head are commercial products, the remaining ones have been appositely designed to meet the systems requirements in terms of portability (small dimensions and lightness) but offering at the same time sufficient stiffness and adequate performance in order to ensure a dimensional accuracy of 0.2 mm on the measurements.

Details on the design of the components and specifications of the commercial products will be discussed on Sect. 3.

3 Procedure Description

3.1 Marker Detection and Pose Estimation

In computer vision, marker detection is a widely discussed topic, which find application in several fields, from industry to gaming [11,12,13]. There are several different types of fiducial marker, each one with its own implemented function library, which allows various operation. Usually both markers and libraries are proprietary and are available under payment. In the current application, it has been used ChArUco fiducial marker [14], with its OpenCV implemented function libraries, which are open access and are usable in MATLAB environment (see Fig. 2).

Fig. 2.
figure 2

ChArUco fiducial marker.

Full size image

In detail, a ChArUco marker is the result of the combination of a black and white chessboard, with ArUco fiducial markers displaced inside its white boxes. ArUco markers are synthetic square markers composed by a black frame and an inner binary matrix which determines its id number, developed by Rafael Muñoz and Sergio Garrido [15]. The copresence of chessboard and ArUco markers allows to benefit of the accuracy in corner detection of the first and the extremely fast detection of the second.

Among the various functions included in the library, the following have been used:

  1. 1.

    Generate marker – to draw the desired ChArUco marker, by defining size of the chessboard grid and the ArUco markers shape;

  2. 2.

    Calibrate camera – to obtain camera calibration parameters by exploiting ChArUco marker;

  3. 3.

    Detect maker – to detect the ChArUco marker landmarks in the picture;

  4. 4.

    Estimate pose – to retrieve the 3D position and orientation of the ChArUco marker wrt camera reference system.

The general procedure, implemented in MATLAB, is completely based on these functions. A graphic user interface has been designed to guide the user during all the procedure. The accuracy of the procedure has been tested under many aspects. Different cameras and lens (see Fig. 3) have been used, as shown in Table 1.

Fig. 3.
figure 3

Adopted cameras: from left to right Canon EOS-M, Fujifilm X-T1, IDS uEye UI 1480 SE.

Full size image
Table 1. Cameras and lens specifications.
Full size table

Each camera has been properly calibrated before each test session. Calibration pictures has been acquired following the procedure suggested in [16], see Fig. 4.

Fig. 4.
figure 4

Some photos of a calibration photo set (IDS uEye UI 1480 SE).

Full size image

At least 32 pictures have been used for each performed calibration.

In order to evaluate the accuracy of the pose estimation phase, the displacement among different marker positions has been calculated by means of the implemented procedure and measured by means of a touch probe (3D Romer Absolute Arm 7520-SI).

Tests have been carried out placing the marker on a calibrated granite plate with a certified planarity error lower than 0.012 mm (grade 0), and on a plywood table with a planarity error of 0.6 mm (support characteristics in the following table) (Table 2).

Table 2. Supports for fiducial marker, used during tests.
Full size table

Since no noticeable difference has been detected by changing camera, only the first stages of the test campaign have been carried out using all the cameras. The following and main tests have been carried out adopting the Fujifilm X-T1 camera. In Tables 3 and 4, some of the obtained results are briefly reported.

Table 3. Test results, calibrated granite plate (Fujifilm X-T1).
Full size table
Table 4. Test results, plywood table (Fujifilm X-T1).
Full size table

Tables show only about 10% of all the obtained measurements. However, the results are significant. Two main observation can be made about them: 1. The overall accuracy of measurements can be considered enough and 2. There are no evidence of advantages in using particularly planar support. In fact, looking at Tables 3 and 4, the errors produced by using a calibrated plate and a plywood panel are approximately in the same range.

As shown in Tables 3 and 4, the results of these preliminary tests show a satisfactorily robustness of the procedure, which is able to obtain accurate measurements with a maximum error in the estimation of the angular position of approximately 1.5 mrad (0.08°).

Tests demonstrated also a slightly influence of support planarity. Thus, no strict requirement on the flatness of the support is required.

On the other hand, the procedure revealed (as predictable) sensible to errors in calibration parameters. Therefore, calibration phase must be carried out carefully and at least 32 pictures are required in order to get a reliable marker pose estimation. More in detail, calibration pictures have been shot by placing camera at a constant distance to the marker (to be maintained during scanning phase). The marker has been placed on the ground, upon a rotating support. In each picture, the marker must cover more than 80% of the frame. At least at three different angular position of the camera w.r.t. ground, the marker is rotated at list in to 8 different positions. Focal length and focus distance of the lens have been maintained during scanning test.

During tests it also emerged that brightness and contrast of the marker in the pictures also influences the accuracy of the detection phase. High contrast particularly benefits ChArUco landmarks detection. In case of low contrast images, corners are not clearly detected, with an unusable result. From this observation, it derives the necessity of provide an adequate lightning for the marker, when the scanning phase happens in dark rooms.

3.2 Laser Line Detection

Based on authors experience and scientific literature, a 4-step procedure has been implemented to detect the laser line in an image, briefly summed up in the following list:

  1. 1.

    Image colour filtering (enhancing laser colour);

  2. 2.

    Background removal;

  3. 3.

    Image filtering with a 2-D Gaussian smoothing kernel (to smooth laser’s speckles);

  4. 4.

    Laser point detection (for each row/column of the image), calculated as the mean point of the Gaussian curve based on pixels laser intensity;

This sub-pixel precision procedure allows the detection of the laser line in the image row by row or column by column, depending on the main laser line orientation.

Colour filtering is applied in order to “enhance” laser colour. Since laser colour is green – only pixels on which green component is prevalent are considered, while the others are neglected. This filter is applied on all the image of the acquisition set (Fig. 5b).

$$ \begin{array}{*{20}l} {I\left( {x,y} \right) \, = \, I\left( {x,y} \right) \, if \, I_{G} \left( {x,y} \right) > \, I_{R} \left( {x,y} \right) \, and \, I_{G} \left( {x,y} \right) > \, I_{B} \left( {x,y} \right)} \hfill \\ {I\left( {x,y} \right) \, = \, 0\;in\;the\;other\;cases} \hfill \\ \end{array} $$
Fig. 5.
figure 5

(a) A test image. Detected reference system is plotted on the bottom-right corner; (b) a zoom of the image of the acquisition set.

Full size image

Successively, background removal is obtained as a subtraction between the image under analysis and the reference image. In particular, the reference image is the last or the first of the entire image set if the considered image is respectively on the first or in the second half of the set.

$$ \begin{array}{*{20}l} {I^{(k)} = \, I^{(k)} - \, I^{(R)} } \hfill \\ {I^{(R)} = \, I^{(1)} \;if\;k\; > \;N/2} \hfill \\ {I^{(R)} = \, I^{(N)} \;in\;the\;other\;cases} \hfill \\ \end{array} $$

The resulting image has an almost black background, on which laser line is clearly visible. However, brightness of the pixels belonging to the laser line is not smoothly distributed: speckles are clearly visible. To obtain a smooth laser line profile and remove noise, a gaussian filter is applied to the image (Fig. 6). In the implemented procedure, gaussian kernel is disk shaped, with a standard deviation equal to 5 (calculated on the basis of the expected laser line width, equal to 20 pixels).

Fig. 6.
figure 6

Illustration of the plotted gaussian curve and the retrieved points.

Full size image

According to [17], laser line is detected by fitting on each row (or each column, depending on main laser orientation) a gaussian curve, considering as x-value pixel coordinate and y-value pixel brightness. In this way, for each row (or column) only a single pixel is considered, and in particular the one that coincides with the centre (mean) of the fitted gaussian curve (see Fig. 6).

3.3 Triangulation

Exploiting information on laser orientation and laser line coordinates in the image, it is possible to proceed with 3D coordinates retrieval by means of triangulation laser camera algorithms [18].

Generally speaking, knowing calibration parameters of the camera (camera matrix and distortion coefficients), the 3D position of each laser line pixel is retrieved as the intersection between the line passing through optical centre of the camera and the pixel and the plane identified by the laser line (i.e. the fiducial marker plane).

4 Hardware

Apart from the camera and its tripod, which are part of the standard equipment of a photographer, the auxiliary system for the acquisition of 3D points is composed by an Y-shaped component that support both laser line generator and orientable fiducial marker board. The support is mounted on a tripod by means of a panoramic head, which allows a horizontal rotation (or vertical, depending on head orientation).

The adopted laser line generator is focusable (i.e. allow to vary the laser focus distance) and emits a 532 nm light wave thus green. This choice is driven by the fact that green light is considered – in scientific literature - easier to detect within digital images. This is due to the fact that SLR camera sensors are generally built on the basis of green dominant RGB colour filters array, e.g. conventional Bayer filter has 2 photosites green out of 4 (50% green, 25% red, 25% blue) and X-Trans has 5 photosites green out of 9 (56% green, 22% red, 22% blue) – see Fig. 7.

Fig. 7.
figure 7

Conventional Bayer filter vs Fujifilm X-Trans.

Full size image

The laser line generator is connected to the support by means of a specifically tailored 3D printed bracket [fig] and is powered by standard AAA batteries (3 V DC - <5 mW).

The planar board for the fiducial marker is jointed to the support by means of 3 countersunk head screws, secured to the planar board by means of bolts. A compression spring for each screw is placed between board and support, providing enough preload with the aim of assuring sufficient stiffness to the joint when board and support are placed a certain distance. Finally, each screw is secured to the support by means of a wing bolt and a spherical washer. Spherical washers have been used in order to assure a proper connection when support and board are not parallel each other (springs are perpendicular to the board, which is orientable w.r.t. the support) (Fig. 8).

Fig. 8.
figure 8

Coupling between planar board and support.

Full size image

The design of both board and support has been developed paying attention to lightness, portability and stiffness. On the light of the relatively low planarity influence of the fiducial marker to the final result, a planarity error of 0.6 mm can be considered acceptable for the board. For this reason, it was cut from a plywood panel. Other solutions have been previously taken into account, e.g. 3D printing by means of FDM technology or sandwich-like aluminium panels, but planarity requirement was not achieved on the first and costs was out of budget on the other. Moreover, FEM analysis demonstrated that the system entirely made from plywood panels (both board and support) is stiff enough to avoid exceeding deformation due to its own weight, in different position. In the worst scenario (horizontal board), a maximum displacement of 0.03 mm was calculated (see Fig. 9).

Fig. 9.
figure 9

FEM analysis on the device: resulting displacements.

Full size image

5 Conclusions

The prototype has been finally tested for the acquisition of the geometry of a painting canvas, in the Confraternita Museum, in Arezzo (Italy), as shown in Fig. 10. The acquisition has been carried out by means of 112 pictures, one for each laser angular displacement. Approximately, laser lines were equally spaced on the plane of the painting, distant around 5–10 mm each other. The resulting point cloud is formed by approximately 200.000 points.

Fig. 10.
figure 10

Test at the Confraternita Museum.

Full size image

This final test verified the accuracy emerged during preliminary tests, carried out in laboratory.

The procedure and the device presented in this paper revealed to be a smart alternative to the cross-ratio based method for the metric acquisition of 3D points. The physical prototype is completely realised by means of inexpensive devices (general cost <200€) and it is easily portable, due to its reduced dimension (A3 format size) and its low weight (<1 kg). The procedure is fast, completely automatic and user friendly. 3D points are obtained directly from the set of pictures, without any user interaction, in a time of around 5–10 min for a set of 100 pictures by using a common notebook. Results have been judged by specialist (Culturanuova) as one step further with respect to the traditional method, both in terms of accuracy and usability.