A Method of Human Facial Portrait Generation Based on Features Exaggeration

Abstract

Caricature is the art which has exaggerated feature without losing identifiable characteristics of true face. People like the caricature and the acceptance level is very high, along with the widely used cartoon. But the cartoon drawing is an artistic behavior and ordinary people has not been trained strictly, so it is very hard to draw a caricature for themselves or anyone else. Based on these, automatic generation system is born to simulate the caricature. However, the difficulty lies in feature exaggeration, which involves extracting features and establishing new exaggeration rules, and the current rules are not unified and integrated. So this paper devotes to establishing effective exaggeration rules, and puts it into practice. This paper takes the Thin Plate Spline to conduct the image distortion in order to achieve the face features exaggeration. Taking the Thin Plate Spline to distort image can achieve smooth effect, and then deformation only for exaggerating features, so it makes the computation easy. The source image after image distortion is slightly “real”, so we use Canny operator to extract face image edge, and use binary to keep texture information, which can generate sketch effect. Finally, a lot of tests demonstrate that the effect of face feature exaggeration is very good.

1 Introduction

The portrait caricature of the human face is the artistic image with both exaggerated style and identifiable features [1, 2]. Because of the exaggeration of the personality features, we can obtain much more useful information to identify objects. So it will be easier to transport more information to identify characters than by photos. Through psychology experiments, Rhodes [3] etc. found that there are enough visual features retaining in the portraitures for face recognition, even they were made up with some single lines. Portrait caricatures are widely used in many areas, like Internet advertising, game, multimedia material, digital mobile entertainment, network chat and so on.

Portrait caricature drawing is a kind of art behavior which is too hard to operate for people who haven’t been trained. The trained cartoonists can well seize character features, exaggerate it in the same direction and create personal portrait which is exaggerated but can be identified easily. The same task is too difficult for people who haven’t been trained, so it’s very necessary for computers to learn this skill to deal with the input face photos and generate caricatures with exaggerated characters automatically. There are many people researching on how to do this, the difficulty lies in the exaggeration of characteristics. However a unified and effective rule of feature exaggeration still did not exist.

In this paper, there is a face portrait caricature automatic generation method based on feature exaggerated. Combined with caricature creation and portrait rendering techniques, we can extract exaggerated features from the frontal facial portrait caricature, and then choose parts of these features for feature exaggerated training. Since exaggerated rules are based on EDFM (Exaggerating the Difference From Mean), we established two benchmark face data models based on different data sources firstly, Normal Face Model (NFM) and Mean Face Model (MFM). Then we summarized the exaggerated pattern of different face component features (the eyebrows, eyes, nose, mouth, face), and determine the features of the various components; the larger the differences between input facial feature with reference facial feature is, the higher the extent of feature exaggeration is; and constraints the “cross-border” situation of exaggerated features based on the proportional relationship, to avoid overlap between some exaggerated face pieces. In addition, features hyperbole is not for all the features, but for a small amount of the outstanding features, automatically select the prominent features is required, firstly calculate the variance, coefficient of variation of the data obtained above, then automatically determined the exaggerated face features according to the results of sorting order. Lastly we should achieve effect of exaggerated characteristics based on the thin-plate spline technology, retain the texture information of the face extract facial image edge and binaryzation method. Finally, edge and texture information of the face will eventually fused into a simple sketch effect face caricature.

2 Research Status

This paper is researching on the generation of face portrait caricature based on the exaggerated characteristics. After inputting frontal face photos, it generates face caricature automatically, which involves technology such as non-photorealistic rendering, image deformation of the face and facial feature extraction exaggeration technology. The simulating of a “hand” style, drawing techniques of producing artistic effects are all belonging to the field of non-photorealistic rendering; Exaggerated effect of face portrait caricature of exaggerated characteristics was generated by corresponding image deformation technology, face feature extraction and description.

In the kind of non-photorealistic rendering of the face image, Sherstinsky [4] designed M-Lattice system which can automatically generate Wall Street Journal-style face engravings; Wong [5] used a semi-automatic method to generate portraits which is closed to charcoal pencil style; Brennan [6] designed and developed a computer portrait generation system, the sketch type caricatures can be generated through the interactive system. These technologies is entirely based on the image, only by physical modeling to mimics the artistic effect of a certain type of art, or did not take the unique physical characteristics of the face into account, it can not generate exaggerated effect.

In the kind of face portraits based on the template, Koshimizu [7] proposed a the exaggerated portraits system based on a template (template-based) in 1999, this system is drawing portrait based on the outline of the entire facial features, because of the line drawing, finally the generated caricature is not very natural, the effect is not very nice. Chen [8, 9] proposed a method based on the example (example-based), the computer could learn many skills of the artist’s painting style and exaggerated features from the huge training data, resulting portraits like of painting style, and made good results, but the rendering time is relatively long, every style imitation of a painter needed to provide a large number of training data. Chiang [10] used the defined model of facial feature points to exaggerate and deform portrait of a painter’s style, and achieved good caricature, but having to manually select points on the painter’s style caricature; Chen Wenjuan [11] proposed a method by separately dealt with exaggerated face shape exaggeration and relationship exaggeration, and made “T rules” of exaggeration and took proportional relationship to describe features of the face and relationship between the characteristics, but finally just generating a simple line drawing.

The existing portraits generation methods have not formed an automatically exaggerating face portraits caricature system. Such a system should be able to automatically obtain the facial feature points, position exaggerated features and determine exaggerated extent, and can also be manually matched to get the facial area. The system from which users can get features exaggerated face caricature by simply inputting their own photos is still in its infancy.

3 Features Exaggerated of Face Portrait Caricature

Exaggeration is one of the main elements of portrait caricature. Applying appropriate exaggerated skills can make a huge impact and shocking on portrait comic visual, and this exaggeration is a conscious, purposeful exaggeration, so there is always some rules existing. When portrait cartoonist draws a face caricature, he starts from the facial feature, and grabs a handful of the most typical features to refine an exaggeration, and exaggerated way is different for different facial feature, so the relative exaggerated extent will be different.

3.1 Face Feature Definition

Through researching and summarizing caricature, the art of painting, aesthetic, ethnological, and in accordance with facial features which is need to be pay attention during caricature drawing on, this article summarizes facial features, especially the characteristics in caricature of hyperbole. The features are divided into a shape characteristic of the human face components (height, width, angle), location characteristics and proportions characteristics between the components.

• (1) Shape features of face

  This article summarizes 32 shape characteristics of individual facial parts, show in Table 1.

  Table 1. Shape features of face

• (2) Relationship between the components of characteristics

  This paper summarizes the characteristics of the relationship between the four parts: eyebrows distance, the inner corner distance of eyes, distance between eyebrow and eye, height gap between tip of the ears and canthus.

• (3) Proportion characteristics:

  This article summarizes the 13 ratio features. In Fig. 1, the 13 ratio characteristics: w₁/w, w₂/w, w₃/w, w₄/w, w₅/w, w₆/w; and h₁/h, h₂/h, h₃/h, h₄/h₃, h₅/h₃, h₆/h₃; w₃/h₂.

  Fig. 1.

  

  Face proportion characteristics diagram.

There are more than 49 individual facial features, and these features are summarized to providing a reference to automatic face drawn studies, and also we can take advantage of these features to complete the search of the face positioning, the positioning of the human face pieces, as well as features related processing.

3.2 Standard Face Model

Each person’s face has their own characteristics, but standard data is not a detailed standard that all the features have a standard provision, but this standard data in the form of proportion covers a lot of features. In accordance with the standard data, this article selected features to do an exaggerated study, proportional value in Table 2 was set based on the above-described face standards.

Table 2. Standard facial feature ratio

This article here in after take advantages of the differences between the characterized human faces and the basic face to achieve exaggeration, normalize these proportions to a certain proportion of a common value, so it is better to determine the extent of the differences, and to select features which required to be exaggerated. According to the proportion of the features of the standard face, and eyes pupil distance has been set as 80 pixels at preprocessing stage, we can calculate the relatively proportion of value of a standard face model as normalized standard, to as a basic for later comparison, the relevant data of the standard human face model as shown in Table 3 below:

Table 3. Standard facial feature value

Now, we have built a standard face model of 12 features which is widely recognized and persuasive, easily obtained. Moreover these 12 features are always noticed and taken corresponding exaggerated characteristics in the cartoons drawing by cartoonists.

3.3 Exaggerated Feature’s Selection and Constraint

Assume that X represents the features of standard face, T represents the features extracted from inputting face. To exaggerate the extracted characteristics, the first step is to determine which features are different from the standard facial features. Then select outstanding differences from different features. At last, we exaggerate features according to the rules of Exaggeration. The difference value Δµ between T and X could be expressed by (1):

$$ \varDelta\upmu = \text{T}_{\text{i}} - \text{X}_{\text{i}} $$

(1)

Features difference is not the absolute comparison of data, but a kind of relative value, to avoid features different being affected by standard data of different features. After standardization process between the difference and the baseline, we can get standard difference value µ:

$$ \upmu = \frac{{\text{T}_{\text{i}} - \text{X}_{\text{i}} }}{{\text{X}_{\text{i}} }},{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} \text{i} = 1,2, \ldots ,\text{n} $$

(2)

n represents the number of features, that is to say n = 12 in the NFM and n = 23 in the MFM. µ could be positive or negative representing different meanings relative to different characteristics of indicators. In terms of the comparison of the length and breadth, the positive value indicates the size needs to be enlarged, the negative value indicates size needs to be reduced; The angle have to be considered in accordance with the actual situation. Generally speaking, the degree of curvature of the eyebrows is an obtuse angle and the average value X_i is negative, so a positive value μ indicates much greater bending angle of the input face of the eyebrows. The size of absolute value of | μ | is show that the size of the gap of the corresponding characteristics between the input and standard facial characteristics. The larger value of the | μ |, the greater the gap, and it means that it is to be much greater possibility of the characteristics to be exaggerating features.

We do not exaggerate all the characteristics in the drawing. The way of exaggerating and the affordable degree are also not the same, so it is a must for us to select the characteristics of being exaggerated and to sort the absolute value of | μ |. The greater the value, the more obvious of the input facial features. The top three features is chosen to be exaggerated in this article.

The face is symmetric, and symmetric characteristics include the width of the left eye and the right eye, the width of the left eye fissure and the right eye fissure, the thickness of the left eyebrow and right eyebrow, the distance form the left eyebrow and eye and form the right eyebrow and eye. These symmetrical features in the average facial model are not exactly the same, but the difference is small. If not treat the asymmetry as a prominent feature of exaggeration, and then consider these features, which means if symmetric characteristic is included in the first three prominent features, then another symmetrical characteristic is also selected to be a prominent feature automatically. In the other words, the width of the left eye is one of the top three, the width of the right eye is also to be prominent features automatically.

The greater the difference between prominent features and standard features is, the greater the exaggerated extent should be.Therefore, the value of the difference Δµ is to be considered in the extent of the exaggerating, a T_i of prominent feature is exaggerated to be eigenvalue T_i′:

$$ \text{T}_{\text{i}}^{'} = \text{T}_{\text{i}} + \text{t} \cdot \varDelta\upmu $$

(3)

After an observation of a large number of people face portrait caricature, we find that different people have different acceptant degree, then it is not the same that the exaggerated extent of the different components. Generally speaking, the facial exaggerated extent will be relatively high, the following is the mouth width, so the exaggeration for the facial feature have an increase on the base of formula (3).

When create a picture of face, it is not only having to consider comic exaggeration, but also ensure the same relative position of facial parts, and not to produce the “overlapping”. So for every exaggerated feature, certain restrictions have to be added for the exaggerated degree. Assuming T_j represents a feature, T_k are shown in Fig. 2:

Fig. 2.

Relationships diagram of the binding characteristics.

For the exaggeration of feature T_k, the constraint is the exaggerated extent is within the range of T_j. For the Fig. 2 (a), both ends length of the T_k are unequal relative to T_j, so T_k will considered to be two parts, for the left exaggerated feature is:

$$ \text{T}_{{\text{k}1}}^{'} = \text{T}_{{\text{k}1}} + \frac{{\text{T}_{{\text{j}1}} - \text{T}_{{\text{k}1}} }}{\upalpha} $$

(4)

The right exaggerated feature is:

$$ \text{T}_{{\text{k2}}}^{'} = \text{T}_{{\text{k2}}} + \frac{{\text{T}_{{\text{j2}}} - \text{T}_{{\text{k2}}} }}{\upalpha} $$

(5)

For the case of Fig. 2(b), there is no need for considering separately for T_k:

$$ \text{T}_{\text{k}}^{'} = \text{T}_{\text{k}} + \frac{{\text{T}_{\text{j}} - \text{T}_{\text{k}} }}{\upalpha} $$

(6)

Here, Set α∈(1,2], in order to guarantee the premise of the characteristics of exaggeration is not out of bounds, and the extent will not to be too small.

The processes of characterized exaggerated described are summarized as follows:

Step 1::

get the difference value Δµ form input facial and reference facial features, and then do standard treatment coming into being μ;

Step 2::

|µ| will be deal with descending sort, and take the first three facial characteristics as subsequent exaggeration.

Step 3::

there is a type of judgment of the facial feature, if one symmetrical features was included, the other feature will to be exaggerated, if one pair of symmetrical patterns was included, take the next facial feature. That is to say, facial features to be exaggerated at least three characteristics, at most five characteristics.

Step 4::

to judge the most prominent exaggerated whether the feature for the facial features or mouth width features, and if were not, calculated exaggerated characterized coordinates in accordance with the degree of aforehand setting, else increased exaggerated extent.

Step 5::

the judgment of exaggerated features is or is not out of range. If it is, get the exaggerated characterized coordinates by way of exaggerating constraint, in accordance with the formulas (4), (5) or (6).

Step 6::

repeat above Step4 ~ Step5, to exaggerate the second most prominent feature, and so on.

4 Facial Comic Drawing Based on the Image Warping

After summarizing some rules of exaggeration, we could complete the effect of the features exaggeration, and by drawing caricature on the deformed image, then generate caricature base on sketch comic style. There are two issues for image deformation—the problem corresponding to the feature points as well as interpolation between the corresponding feature points. Deformation of the face image, we would like to do some local deformation according to the facial constitution, especially the exaggeration of significant features that it is required in the article. Good local deformation is exaggerated characteristics of the area in need of an image deformation, but also has some smooth influence for the surrounding area, so that the image does not produce the effect of folds or faults. The thin-plate spline deformation technique is selected to complete the feature exaggeration in this article.

4.1 Thin-Plate Spline Deformation Technology

The thin-plate spline function (Thin-Plate Spline, TPS) is a radial basis function, which is based on the elastic deformation of the point constraints proposed by Bookstein [12]. This function is mainly used to solve the problem of interpolation between the two set of image points. A conversion function is established between feature points in the two images to match corresponding feature points, and to make the total energy of the image deformation controlled to be the minimum. Namely, to ensure an integral deflection minimize norm [13].

In this article, we get 78 face features points and coordinates value of all feature points. Through selection of feature exaggeration and determining of exaggeration degree, the coordinate position of the corresponding feature point in the target image can be obtained, and thus can determine the respective coefficients the source image mapping transformation to the corresponding feature points of the target image, and then the source distance input deformation algorithm of the feature point with other feature points of the features of the image to be deformed, and the coordinate value of each corresponding point in the target image can be obtained. In other words, the feature point coordinates in the source image and feature point coordinates in the target image could be calculated by way of above formulas to generate new coordinates in the target image.

In order to show better experiment results, the parameter values can be set manually to make the extent of deformation relatively large, and the deformation is not characterized “same direction” exaggeration in the Caricature Creation. Shown in Fig. 3, the change in length of the eye does not change the binocular distance and the change of the eye fissure width does not change facial distance. The deformation technology based on thin-plate spline function effectively achieve exaggeration of features deformation, and make deformation’s effect smooth, not produce folds and distortion effects.

Fig. 3.

Exaggeration of eye shape features.

4.2 The Synthesis of Man Face Portrait and Comics

The facial feature information can be expressed in two parts: contour information and texture information, the former represents the facial macro-structure features, the latter represents local detail features. So the two parts can be deal with separately, and later make a superimpose, get sketch portrait finally. The contour can be get by Canny operator, texture information would be obtained by binary image. In this paper, firstly, Canny operator is used to deal with edge detection. Secondly, use binary image to preserve the texture of the face. Finally, combine the texture with the edge extracted form Canny operator to generate a sketch comic effect, as shown in Fig. 4.

Fig. 4.

Facial caricature synthesis.

5 Experiment and Discussion

The flow chart of exaggerated caricature algorithm based on the characteristics of the standard face is shown in Fig. 5.

Fig. 5.

Flowchart of caricature’s generation.

Standard face is obtained in accordance with the recognized standard face proportional value, extracting facial feature points from the new input face images. In accordance with the requirements of this article, we can obtain 12 facial features (standard face model) in further processing, and then select the prominent features in accordance with exaggerated rules of this article and exaggerate features in accordance with the constraint of exaggerated degree.

The experimental effect is shown in Fig. 6:

Fig. 6.

Experimental effect diagram.

6 Summary and Expectation

The paper generate a kind of facial feature exaggeration based on recognized popularly standards face, and the data of this widely recognized model are obtained conveniently. For getting 12 exaggerated features, we only need 19 individual facial feature points. Feature exaggeration could be achieved by thin-plate spline deformation technology, face edge Information is retained by the use of Canny operator simultaneously, then we can reserved facial texture information by using the binary image, and eventually get sketch effect of caricature through the fusion of edge and texture information.

The follow-up study included the exaggeration of a side face. Preprocessing the images of the training set and a user input image are used manual method to locate the eye pupil, the automatic positioning of the pupil will be also completed in follow-up work.

The subject of the facial feature exaggerated is considerable difficulty, so there is not a kind of efficient and robust facial feature exaggeration system in this area. The aim of this paper is to raise some more rational thinking, get some desirable progress on some of the major part of the solution, and have a certain contribution and reference value to learning and application.