Upper Layer Extraction of a Folded Garment Towards Unfolding by a Robot

Abstract

A layer extraction method aiming at robotic garment unfolding is presented in this paper. Utilizing depth sensor input, the layer detection is achieved through a depth first algorithm and a simple perceptron. An action space based on the edges detected on the garment and their features restricts the complexity of the algorithm and accelerates the detection. This paper constitutes part of the robotic garment unfolding pipeline and emphasizes on the extraction of the upper layer of clothing articles that result in a half folded, two-layer planar state after robotic manipulations. The presented methodology is independent from the type of the handled garment and can cope with unknown designs. First results seem promising and encourage future work on the field.

1 Introduction

Home service robotics is a scientific area recently receiving augmented attention in the roboticists’ world. A challenging task that robots have to face while doing housework is the manipulation of pieces of clothing. The main challenge of these tasks lays on the fact that clothing articles are highly non-rigid items with infinite degrees of freedom hence it is difficult to manipulate them and define their form.

Over the last years, robotic research groups have been occupied with the robotic unfolding of clothing articles. Several methods approach the unfolding task using an intermediate step where the garment is brought to a half folded configuration that facilitates the further unfolding [1,2,3,4] whereas other approaches detect characteristic features that, when grasped, lead to the natural unfolding of the clothing article due to gravity [5,6,7] (e.g. a T-shirts shoulders).

This paper proposes a two phase procedure for the task of robotic garment unfolding: (1) bringing with robotic manipulations the clothing article in a half-folded, two-layer state, as it is presented in [4], (2)extraction of the upper layer of the garment. In the second step, depth information analysis of the half-folded garment provides a set of edges that includes segments of the garment’s upper layer and noise. Thus, a search strategy utilizing a simple perceptron and a depth first algorithm explore the edges to obtain the sequence forming the upper layer.

The main contribution of this method is that it is independent of the garment’s type and shape. This property allows handling different kinds of clothing that sometimes might have unusual shape. For example a woman’s wardrobe might have irregular garments according to fashion, e.g. a shirt with only one sleeve or an uneven hemline. Contrary to other methods that handle only specific types of garments known a priori, our method can cope with unknown designs. Furthermore, it does not need training with large databases nor uses predefined templates.

2 Bringing the Garment in a Half Folded Planar Configuration

In this paper the unfolding procedure is considered a two stage task. In the first part the garment is brought from a random, crumbled configuration in a half-folded, planar state whereas in the second part, the two layers in which the garment is divided are extracted so that it can be completely unfolded.

The procedure of transforming a piece of clothing from a random state to a half folded planar configuration was analysed in detail in [4]. The garment, which was initially lying crumbled on a table, was grasped by a random point by one robotic manipulator and rotated in front of a range sensor acquiring its depth images. Each depth image was analysed so that folds and hemline edges were extracted. Although a single picture of the garment could have been sufficient to extract folds, aggregating results over more viewpoints improved the robustness of the algorithm. Therefore a voting procedure that designated the best candidate fold was utilised facilitating the avoidance of erroneous folds. Once the first fold was grasped by the robot the aforementioned procedure was iterated so that a second outline point was detected and grasped. Finally, the garment was located half-folded on a table to proceed with further unfolding.

3 Detection of the Garment’s Upper Layer

In this section a method for the extraction of a half folded garment’s upper layer is introduced. We divide the procedure in two parts: (1) the extraction of edges formed inside the garment and by its outline and their conversion into straight, oriented edges so that they are easily managed, (2) the interconnectivity of these edges in order to extract the upper layer of a half folded piece of clothing.

3.1 Extraction of Straight, Oriented Edges

The first step of the proposed method is the acquisition of a depth image of the half folded garment using a range sensor (Asus Xtion). The image is preprocessed by means of bilateral filtering aiming at reducing texture and noise and enhancing the edge contours extracted by Canny detector. Since our goal is to transform the edge contour pixels into separate edges, we cluster all the garment’s internal pixels using the dbscan algorithm (the outline of the garment is considered to be a separate cluster). Subsequently, we apply line simplification to all the clusters so that they result into straight line segments. In order to diminish the noise, all the small edges are rejected.

Fig. 1.

The edges’ orientation according to the direction of the path’s exploration

In order to attribute orientation to the detected straight edges, the depth difference between the two sides of each edge is calculated. In particular, a rectangular area around each edge and parallel to it is examined. The edge separates the area in two equal parts and the average depth of each part is calculated. The edge’s orientation depends on the direction of movement while exploring possible outlines of the upper layer. Thus, when the direction of the exploration is made clockwise the upper layer is supposed to lie on the right side of the edge, while in the opposite exploration direction the upper layer is supposed to be on the left side (Fig. 1). Therefore the orientation of an edge with two endpoints p and q is:

$$\begin{aligned} \frac{\overrightarrow{pq}}{|\overrightarrow{pq}|} = \overrightarrow{u} \times \overrightarrow{z}, \end{aligned}$$

(1)

where $\overrightarrow{z}$ is the unit vector of the rotation axis determining the exploration direction and $\overrightarrow{u}$ is unit vector which is vertical to the edge and its direction occurs from the depth difference between the two sides of the edge.

3.2 Upper Layer Extraction

The edges that occurred from the procedure described in the previous section belong either to parts of the garment’s layers or to wrinkles. The existence of wrinkles, the gaps and the curvature of the garment’s edges compound the difficulty of the layer extraction problem. The goal of the proposed method is to find the sequence of edges that connects the a priori known grasp points, that brought the garment in the half folded configuration, and forms the outline of the garment’s upper layer. In order to determine this sequence a search algorithm is defined. The algorithm is defined by the tuple $\langle E, A, T\rangle$ and actually comprises: (1) a state space E, since the exploration of the sequence proceeds based on each edge’s features, each edge determines the sequence and is considered a different state, (2) an action space A, that comprises the possible interconnections between the states (3) a state transition function T, that based on the states suggested by the action space determines the subsequent state of the sequence and (4) a search procedure, that facilitates handling of dead-end sequences, i.e. sequences that do not connect the two grasp points.

State Space. The state space E is defined as the set $E= \{s_1,s_2,...,s_N\}$, where N is the number of detected edges, excluding the edges between the two grasp points that constitute the folding axis to avoid taking this short-cut as a candidate path. The goal is to find the set $E_L \subset E$, i.e. the edges of the upper layer.

Before analysing the path extraction procedure the initial and goal state of the path are determined. These states are based on three types of folded upper layers that might occur in reference to the edges that start from the folding axis. In particular, the starting and ending edges of a layer might be: (1)both inside the outline of the half-folded garment, (2) one inside and one located at the outline, (3) both located at the outline. Thus, the initial state $s_1$, i.e. the starting edge of the path, is detected by evaluating the edges based on their orientation and location in relation to the grasp points. In particular, if there is more than one internal edge with orientation starting from one of the grasp points (i.e. the line connecting the starting grasp point and the start of the edge should be collinear or form a very small angle $\omega$ $< th$, where th is a threshold), then, the nearest edge is selected while when there is no internal edge with such orientation, one of the outline edges starting from the grasp points is chosen. The preference to the internal edges as parts of the upper layer’s outline is justified by the fact that in this case an “arrow” junction is implied, i.e. the fold that created the layer [8]. In the same spirit, the goal state $s_G$, i.e. the final edge of the path, is found. It is either an internal edge oriented towards the final grasp point or an outline edge resulting to it.

Fig. 2.

Four types of connection: (a) collinear edges, (b) edge with inclination towards an outline corner, (c) continuous edges, (d) proximate edges.

Action Space. The action space $A=\{a_1,a_2,...,a_n\}$ defines the n possible state interconnections as they occur based on certain criteria. Thus, the exhaustive examination of all the edges is avoided and only certain actions are available at each state of the path. The criteria concern the edge’s collinearity, contiguity, proximity and inclination. In particular, we distinguish four types of connection (Fig. 2):

• 1. Collinear edges (Fig. 2a). It is common in perceptual completion to prefer smooth interpolating curves of least energy.

• 2. Edges with inclination towards an outline corner (it is an indicator of a non accidental relationship that connects an internal with an outline edge, Fig. 2b).

• 3. Continuous edges, which are mostly outline edges (Fig. 2c).

• 4. Proximate edges (Fig. 2d), meaning edges whose intersection point is located inside the garment’s outline, the line connecting their ends does not intersect with the outline (e.g. one edge is on the sleeve and the other on the main body of a shirt) and their distance is smaller than half of the outline’s length.

Fig. 3.

Angles utilized as perceptron inputs

It should be mentioned that in all the cases described above the edges are oriented.

State Transition Function. The state transition function $T(s_i,(a_i1,..,a_in))=s_i+1$ is a recursive perceptron that compares two possible states at a time, as they occur from action space $A=\{a_{i1},..,a_{in}\}$ and selects the final candidate $s_{i+1}$ for the path. To proceed to this stage, the branching part described in the previous paragraph has to be implemented for all the candidates, i.e. we proceed one step further exploring the possible subsequent edges. In this way a better understanding of each candidate’s position in the sequence is obtained. The important features of each candidate that affect the selection of the path’s components and are used as inputs for the perceptron are:

• 1. The connection type t, where $t={1,2,3,4}$ and corresponds to the numbers given at the description of the types at the previous paragraph.

• 2. The absolute value of angle $\theta$ (Fig. 3) needed to make the candidate edge $s_{i+1}$ collinear with the current edge $s_i$. This feature indicates the abrupt changes of the path’s curvature that might occur from false inclusion of wrinkles.

• 3. The absolute value of angle $\varphi$ (Fig. 3) needed to make the subsequent of the candidate edge $s_{i+2}$ collinear with the candidate edge $s_{i+1}$. The goal of this feature, which is similar to the previous one, is to perceive the position of the candidate edge according not only to its previous but also to its subsequent edge.

• 4. A binary input that shows whether the candidate edge $s_{i+1}$ is internal or external in correspondence to the outline of the folded garment. The existence of edges in the garment’s interior is an indication that they are parts of the garment’s upper layer and are preferred over the outline edges.

• 5. A binary input depicting whether the subsequent of the candidate edge $s_i+2$ is internal or external.

• 6. The distance d between the current edge and the candidate. Since the edges are oriented, the distance is defined from the end point of the current edge and the start point of the candidate edge. The smaller the distance the more possible is for the edges to be connected.

Fig. 4.

Three different options provided by the action space and encountered by the perceptron.

Fig. 5.

Outline of layer extraction: (a) the half folded garment, (b) the extracted sequence of edges that form the layer, (c) the extracted layer with the detected edges connected.

In Fig. 4 the action space provides three different options of states. As it can be seen in Fig. 5, the transition function via the perceptron resulted in the correct outcome.

Search Procedure. The state transition procedure returns a preferred edge that constitutes the next edge of the path. Nevertheless, sometimes this might result in a path that does not lead to the final grasp point or a path that includes a loop. Both of these cases are rejected, as a result the searching for another path should start. To deal with this situation we use a depth first algorithm. Therefore, we return back to the last node where there was a branching, discard the edge that led to dead end and select the next edge that the state transition function suggests. This procedure is iterated until a path that results to the final grasp point is found or until there are no other branches to follow.

Fig. 6.

Examples of extracted layers

4 Results

The depth images used for the method’s evaluation were acquired by an Asus Xtion range sensor, located at a fixed height from the working table. The garments were manually placed onto the table but they were not flattened. Since the very thin garments’ width is smaller than the range sensor’s tolerance we used only thick articles such as thick long sleeved shirts and pairs of pants.

For the perceptron’s training 17 branching examples were used that occurred in 12 different configurations of 3 garments (two long sleeved shirts and a pair of shorts). As a testing set 30 cases of branching were used that occurred from 18 random configurations. In 27/30 of the cases the perceptron’s output was the correct.

For the evaluation of the whole method 34 different configurations of 5 articles were used (three long sleeved shirts and two pairs of shorts). In 27/34 the method detected correctly the upper layer (in 4 cases there was a deviation due to lack of edge segments representing a whole edge of the garment’s upper layer polygon). In 4 cases the result was wrong due to the perceptron’s outcome, in 2 because the garment was too wrinkled for the branching procedure to gather the correct candidates and in 1 case a big part of the upper layer was not visible, i.e. there were no edges representing it. In Fig. 6 examples of extracted layers in different types of folded garments are depicted.

5 Conclusions

In this paper, a layer extraction methodology, part of a robotic garment unfolding pipeline, is introduced. The method, utilizing depth data, analyses the detected edges on the garment’s surface and according to their features and interconnections extracts through a depth first algorithm and a perceptron the edge sequence that comprises the layer. In particular, the presented method focuses on the detection of the upper layer of garments that result in a planar half-folded state due to robotic manipulations. The leverage of the proposed procedure is that it remains independent of the garment’s type handling various garment designs. Since the results provide a high rate of success, future work testing the method in an extended dataset is planned. Furthermore, a methodology that determines the appropriate grasp points on the detected layer combined with the planning of the robotic movements to achieve successful unfolding is being developed.