Keywords

1 Introduction

Generally, researchers about robotic manipulation have been focused to recognize rigid objects such as solids [1, 2]. But, in recent years, the manipulation process has changed to recognize articulated objects [3], deformable objects [4] such as soft objects [5] semi-solids such as organic material [6] or tissue [7]. Therefore, three basic types of rigid objects can be considered solids, elastics and deformable objects.

The object rigidity can be mathematically measured with three different interpretations: stiffness which is dependent on the force and the size of area where it is applied, hardness which defines the forces required to penetrate the material and toughness which is the amount of energy that a material can tolerate before it can be fractured.

Using tactile and/or force control is usually to manage the grasping processes [8]. Thus, a robot hand is able to do task manipulation of objects applying forces that is not usually enough large to break the surface structure or to drill it. In a rule, the forces never exceed a value which can cause a rupture or penetration. Notwithstanding, this is a complex task in which just the tactile control does not allow us to avoid deformations. More data are required to control the manipulation process if the deformations wish be controlled and measured [9]. Sometimes, the tactile and force information is poor, inconsistent or ambiguous to detect and analyse deformations in an object which is being manipulated [10]. In contrast to [11], the goal of this paper is to model and identify deformation in semi-solids with elastic properties from contactless sensors. Thus, visual sensors can assist to other sensors [1, 12, 13]. Firstly, they built a geometrically modeled object and secondly, they identify the object deformation for comparison between points of surface with and without deformation.

Consequently, the elastic object can be construed just as a stiff object because there is not surface penetration or rupture of its structure. The elastic modulus has often been used to measure the stiffness properties in the study of materials when they are known. The elastic modulus measures the applied force per unit area to deform an object surface. However, the elastic modulus cannot be used when the objects were made with an unknown material. In this paper, the stiffness can be measured by the curvature in surface points from the object geometry. The curvature features are computed from mesh points which models the object surface.

The paper is structured as follows: the concept of curvature to measure curves by means differential geometry is shown in Sect. 2. In Sect. 3, we present the method based on surface variation to model surfaces, detect and measure deformation in those. Experimental results of the deformations caused in virtual robot manipulation tasks are shown in Sect. 4. Finally, Sect. 5, contains the conclusions.

2 Mathematical Approach to Compute Curvature of Surfaces

In this paper, the differential geometry of curves is used as a tool to propose a method which allows us to analyse 3D surfaces. Generally, in computer vision, the 3D object surface consists of an unstructured point cloud represented as \(\mathbf{P}=\{\mathbf{p}_i \in \mathfrak {R}^{3}\}\). If a plane object in the Euclidean space is deformed by pressing on its outside surface, the geometric properties change and some smooth curves could appear. Here, the curves are understood as a variation of surface, and they can be computed by measuring of the orientation change of normal vector to the surface.

Generally, each point \(\mathbf{p}_i \) satisfies a set of axioms in relation with its neighbourhood environment. Thus, the computation of the geometric properties of the curves depends on the k-nearest neighbour points to each \(\mathbf{p}_{ij} \). Therefore, \(\mathbf{P}\) can be sampled as a set of patch \(\mathbf{N}_j \) and each one is a subset of points \(\mathbf{p}_{ij} \). Both, the size (radius) and number of points of a patch (dense) influence the accuracy to compute the geometric properties of the curves. Few points cause inaccuracy. In contrast, many points distort the values hindering the detection of orientation changes in the surface. In this last case, the detection is smoothed.

An analysis of the eigenvalues computed from covariance matrix of points in a neighbourhood environment according to (1) can be used to estimate the local geometric properties of a patch of the surface [14]. To define the covariance matrix is applied PCA (Principle Component Analysis) as follows:

$$\begin{aligned} {\mathbf{C}}_{\mathbf{P}} ={\mathbf{PP}}^{T}=\left[ {{\begin{array}{cccc} {{\mathbf{p}}_{i1} -{ \overline{\mathbf{p}}}} \\ \cdots \\ {\mathbf{p}_{ik} -{\overline{\mathbf{p}}}} \\ \end{array} }} \right] \left[ {{\begin{array}{cccc} {\mathbf{p}_{i1} -{ \overline{\mathbf{p}}}} \\ \cdots \\ {\mathbf{p}_{ik} -{\overline{\mathbf{p}}}} \\ \end{array} }} \right] ^{T} \end{aligned}$$
(1)

where each \(\mathbf{p}_{ij} \) is a point of the neighbourhood environment \(\mathbf{N}_j \) and \({\overline{\mathbf{p}}}\) is the centroid of the patch. And k defines the number of points \(\mathbf{N}_j \).

Equation (1) is solved by Turk and Pentland method [15] that allow us the computation of eigenvalues and eigenvectors of \(\mathbf{C}_\mathbf{P} \) with low computational cost by building a matrix \(\mathbf{A}=\mathbf{PP}^{T}\) and by applying singular value decomposition (SVD) to \(\mathbf{A}\) as follows:

$$\begin{aligned} \mathbf{A}\cdot \mathbf{v}_j^{\prime } =\lambda _j^{\prime } \cdot \mathbf{v}_j^{\prime } \rightarrow \mathbf{P}^{T}{} \mathbf{P}\cdot \mathbf{v}_j^{\prime } =\lambda _j^{\prime } \cdot \mathbf{v}_j^{\prime } \end{aligned}$$
(2)

where \(\mathbf{v}_j^{\prime } \) are the eigenvectors and \(\lambda _j^{\prime } \) are the singular values of the matrix \(\mathbf{A}\). Multiplying by \(\mathbf{P}\) is obtained:

$$\begin{aligned} \mathbf{PP}^{T}{} \mathbf{P}\cdot \mathbf{v}_j^{\prime } =\lambda _j^{\prime } \cdot \mathbf{Pv}_j^{\prime } \rightarrow \mathbf{C}_\mathbf{P} \mathbf{P}\cdot \mathbf{v}_j^{\prime } =\lambda _j^{\prime } \cdot \mathbf{Pv}_j^{\prime } \end{aligned}$$
(3)

Then, the eigenvalues and associated eigenvectors of \(\mathbf{C}_\mathbf{P} \) can be obtained as:

$$\begin{aligned} \lambda _j =\lambda _j^{\prime } \hbox { and }{} \mathbf{v}_j =\frac{\mathbf{p}\cdot \mathbf{v}_j^{\prime } }{\sqrt{\lambda _j^{\prime } }} \end{aligned}$$
(4)

The eigenvalues sum provides information about the surface variation between each point of the patch \(\mathbf{N}_j \) and its centroid. In addition, the smallest eigenvalue provides a measure of the variation along the normal vector to the surface. Consequently, the eigenvalues can help to classify the concavity of the surface at each point, and the set of all defines the curvature parameter. Thus, the local maximum curvature [16] of \(\mathbf{p}_{ij} \) within the patch \(\mathbf{N}_j \) can be computed by:

$$\begin{aligned} c_{ij} =\frac{\lambda _0 }{\lambda _0 +\lambda _1 +\lambda _2 } \end{aligned}$$
(5)

where \(\lambda _0 \le \lambda _1 \le \lambda _2 \) are eigenvalues of \(\mathbf{C}_\mathbf{P} \). The associated eigenvector \(\mathbf{v}_0 \) is the normal vector to the tangent plane of the patch surface \(\mathbf{P}\) that define \(\mathbf{v}_1 \) and \(\mathbf{v}_2 \). The set of level curves is defined as a function \(\mathbf{S}_\mathbf{P} :\mathfrak {R}^{3}\rightarrow \mathfrak {R}\) as follows:

$$\begin{aligned} \mathbf{S}_\mathbf{P} (\Phi )=\left\{ {\left( {x,y,z} \right) \in \mathfrak {R}^{3}:\Phi \left( {x,y,z} \right) =l} \right\} \end{aligned}$$
(6)

where l is a constant value and it represents a level curve in the surface. Then, every level curve is computed like a cluster with the same colour that represents the points of \(\mathbf{P}\) with a similar value of curvature (Fig. 1).

Fig. 1
figure 1

Level curves set \(\mathbf{S}_\mathbf{P} \) computed from a surface \(\mathbf{P}\) with a deformation

Full size image

3 Our Method to Find Surface Variations

In this work, the surfaces are represented as a point clouds. It is worth nothing that the data of \(\mathbf{P}\) can be obtained from a structured or unstructured way. \(\mathbf{P}\) is structured whether it can be stored in a matrix \(\mathbf{P}_{X\times Y} \) where X and Y are the number of rows and columns, respectively. This occurs whether \(\mathbf{P}\) is acquired from a sensor like a RGBD or ToF, then each point corresponds with a position of the sensor. In contrast, \(\mathbf{P}\) is unstructured whether it can only be stored as a vector matrix \(\mathbf{P}_{X\times 1} \) . In this last case, \(\mathbf{P}\) has no direct relationship with the data of range image. \(\mathbf{P}\) is usually unstructured when it is built from a virtual CAD model (Fig. 2a). Consequently, in order to find the neighbor of each point of \(\mathbf{P}\) is typically used either kdtree or octree data structure.

Fig. 2
figure 2

a CAD Model of a deformation caused in a planar surface by contacting of two different geometric objects: Sphere and Cube. b Level curves set computed from the deformation and represented with a different colour l. c Histogram

Full size image

The proposed algorithm has two phases: Initialization and Extract-Curvatures. The first step is only computed when \(\mathbf{P}\) is unstructured because it needs to be structured to get sorted points. The closest points in the Euclidean space must be stored as neighbors within \(\mathbf{P}\). Thereby, it is possible to search points with the same geometrical properties and located in the same neighbourhood environment. This is essential to calculate the level curves from the curvature parameter and to detect surface variations given by the transversal paths to the level curves.

The proposed method detects deformations of any surface represented as \(\mathbf{P}\). It is based on finding the critical points of \(\mathbf{P}\) in which there are surface variations. In this work, the critical points are points belonging to different level curves \(\Phi \) of the surface but also, they lie in the transversal path that goes across the level curves by fitting of singular points. The singular points are defined as the points with maximum curvature values, and whose curvature value is estimated like a curvature threshold computed from a Histogram of curvature.

3.1 Histogram of Curvatures

Once the curvature parameter, according to (5), is calculated for all points within \(\mathbf{P}\) (Fig. 2b), a curvature histogram is built (Fig. 2c). It represents the distribution of the surface variation and it can be computed as follows:

$$\begin{aligned} H_\mathbf{P} =\frac{\text {Number}(c_{ij} )}{\text {size}(\mathbf{P})} \end{aligned}$$
(7)

where \(c_{\textit{ij}} \) is the curvature value, and \(\text {size}(\mathbf{P})\) represents the density or number of points used to sample the surface. \(H_\mathbf{P} \) changes depending on the size of the neighbourhood environment \(\mathbf{N}_j \) used to compute each \(c_{\textit{ij}} \). The user must choose the radio according to the accuracy for the detection of abrupt changes in the surface (Fig. 5).

The histogram \(H_\mathbf{P} \) is useful to find the singular points, that is to say both the points with \(\max (c_{ij} )\) and the boundary points which define the border where there is no curvature variation in the surface. The boundary points split the curvature region and the non-curvature region. They are computed from \(H_\mathbf{P} \) using the technique presented in [17]. The histogram allows us to find the curvature threshold \(\varepsilon _{\textit{otsu}} \) by minimizing the standard deviation of the Gaussian distributions that represent the two zones (Fig. 2c).

figure a

3.2 Path of Critical Points to Measure the Surface Variations

We assume that the surface is always differentiable, then the surface gradient at a point is either zero or perpendicular to the level curve which represents the surface variation at that point. We inspect the critical points of the surface function along its level curves. Our critical points are the curve points which follow the direction of gradient.

The proposed algorithm is used to find the critical and boundary points and the transversal path which define the deformation on the surface. The algorithm uses an octree O like [18] in order to contain all points p of \(\mathbf{P}\) as a sorted structured. Later, we create a priority queue Q of pairs \(\left\{ {p,N} \right\} \), where p is each point of O and N is its adjacency list. N represents the 26-connected neighbour points. Q puts on the head, the points p with the greater curvature values. Later, Q is filtered to only obtain the values of Q which are greater than \(\varepsilon _{\textit{otsu}} \) then they are stored in R. Subsequently, R is crossed according to algorithm from line 18 until 24 in order to get a list with the deformations from the clusters D. Finally, we use Dijkstra’s algorithm \(\text {FindMinPath}(v_1 ,v_2 )\) to find the minimum paths. Let \(v_1 \) be the point with maximum curvature (the head into each \(D^{\prime })\) and let \(v_2 \) be the target point (points into the boundaries).

4 Experiments

Several experiments and tests have been done. We have modelled the movements of a real robot hand considering both the kinematics and the virtual model (shape and structure) of their fingers and palm without physical constraints and without considering singularities, by using free software Blender. In particular, the model of robot hand corresponds to a Shadow Hand Robot available in our research laboratory.

4.1 Experiments to Compute Curvatures Skeleton

The deformations are described by curvature skeletons. A curvature skeleton is a set of minimum transversal paths. These path goes across the level curvatures, thus they start in the same critical point (maximum curvature) and end over the boundary points. An example of the skeleton is represented in the curvature map of the Fig. 3. The chart of that figure shows the minimum transversal paths which define the behaviour of the deformation. Thereby, if all paths have similar distances then the deformations are homogeneus (in this case, the values are all close to 0.1 m and then the deformation shape could be understand like a circle). In opposition, if the paths represent different distances then the deformation is heterogeneous (in this case, other irregular deformation shapes can be read). Furthermore, the chart shows the deformation as a set of branches which represents several minimum transversal paths. The number of branches and the value of slope of them determine how is the deformation: steep, smooth, flat, etc. In Fig. 3, there are two groups of path (group 1: path 1, 2, 4, 6, 8, 11; group 2: path 3, 5, 7, 9, 10, 12) and each one tends to curvature values around to 0.03 (group 1) and 0.005 (group 2).

Fig. 3
figure 3

Characterization of the transversal path variations computed from critical points

Full size image
Fig. 4
figure 4

Temporal evolution of the deformation of a planar surface caused by contact of a robot finger from two movement sequence of the robot hand. a Test 1. b Test 2

Full size image
Fig. 5
figure 5

Evolution of the curvature mean value through the movement sequences shown in Fig. 4

Full size image
Fig. 6
figure 6

Characterization of the transversal path variations computed from critical points for the last frame of the Test 2

Full size image

4.2 Experiments Applied to Grasping Tasks

Our experiments simulate several deformations which are caused by contact between fingers of the virtual robot hand and a planar object. The deformations depend on the orientation of the fingers, the trajectory used in the contact process, the shape of the finger and the pressure produced by it when the contact occurs (Fig. 4).

Each experiment consists of a movement that shows an image sequences. Each sequence is done from 60 images for testing. The algorithm computes the level curves set from both the curvature parameter and the histogram of curvatures. They allow us to obtain the evolution of the curvatures during a time sequence through the formation process of the deformation (Figs. 4 and 6). Afterwards, we estimate the path of critical points to measure the surface variation and the topographic profile of the deformation (Figs. 5 and 6).

The algorithm is implemented with the C++ language and using the open source libraries PCL, Boost and Eigen. It runs over a computer with a Core i7-4770k processor, equipped with 8GB of system memory and an nVidia GeForce 760GTX.

Figure 5 shows the dependence of the curvature measures when the radius of the neighborhood is chosen in a proper way or not. If the radius is not enough large then the smallest gaps on the surface are not detected by the algorithm. Also, Fig. 5 shows the evolution of the boundary points and how they grow quickly in the first iterations. This fact indicates that the contact between robot hand and elastic surface is occurring in that moment. Later, the growth is smoother and more progressive. The curve slope indicates the deformation level generated during the grasping over time. Figure 6 shows us how the minimum transversal paths retrieve us information to determine the deformation in an instant time. In the test 2, there are two dominant paths 2 and 3 because they are far from the rest of paths (those other are close and have similar length). The paths 2 and 3 determine the shape of the deformation caused on the surface. But also, the paths 0, 1, 4, 5 and 6 show a sharp slope which represents an abrupt deformation. This is much variation among level curves located near on the surface.

5 Conclusions

This paper has described a novel approach to analyse the deformation of flat surfaces. The proposed method builds curvature variation maps to measure the local unevenness in the surface by comparison among the points that lie on the same surface. Also, the method uses a novel algorithm to characterize and to describe the curvature variation caused by deformation. To do it, our algorithm identifies the level curves and later, it finds the critical points which define transversal paths among those level curves. Thus, we generate topographic profiles in the local maximum gradient directions. The gap, between the curvature value and the Euclidean distance of the critical points, determines whether or not deformation and the degree of slope that defines if the deformation is smooth or abrupt.

The results reveal that this surface analysis provides an empirical investigation on the effect of the deformations about the surface of elastic objects when several grasping tasks with robot hand are performed. And most importantly, an algorithm to measure these deformations has been implemented and tested successfully. It is programed in C++, and it can be integrated into robotic platforms.