Keywords

1 Introduction

The earliest origin of the term neural art is in Gatys [1]. This paper is different from the traditional image style artistic method. It pioneered the use of convolutional neural networks to learn oil painting style. And Gatys also detailed the steps of image style transfer based on VGG19, which described the formation of style expressions, texture combining methods, and loss functions in more depth [1,2,3].

Compared to Gatys using Gram matrix to calculate the overall style effect, Li proposed to use the local texture of the style image for style transmission [4, 5]. Ulyanov, proposed a texture combining algorithm based on feedforward network. Since the network is trained in advance, their algorithm execution is faster than the Gatys algorithm [6]. Moreover, some people have designed a more Fast networks need to be trained on some large datasets such as COCO datasets, and then applied to existing images that need to be artistic, which greatly saves training time [7]. Recently, Goodfellow et al. [8] proposed a Generative Adversarial Network (GAN). Deep neural networks based on GAN ideas are gradually applied in the field of image style transfer [9,10,11]. For example, CycleGAN [12], DiscoGAN [13] and DualGAN [14] use the cyclic consistency loss to save the key information of the input and transformed images, and realize the image style transfer in the case of unmatched data.

Gu [15] proposed a feature reorganization style migration method based on vgg neural network. They link most parametric and non-parametric style migration methods by readjusting style loss. It has a good effect for general images. But not effective to Chinese painting, because Chinese painting is different from natural images and has special style characteristics. Therefore, this paper try to extract these features and use them to improve the original method to improve the effect of Chinese painting.

2 Style Description of Chinese Painting

Chinese painting is one of the oldest continuous art traditions in the world. It is derived from calligraphy, which is essentially a linear art, using strokes to evoke pictures and emotions. The painting techniques of Chinese painting are mainly expressed in four aspects: pen, ink, color and composition [16].

The use of pen is the essence of Chinese painting. Chinese painting is line-shaped and reaches its peak in the use of line technology. Thickness and thinness of the line, long and short, thick and light, light and heavy, and virtual and real are all different things. Important factors [17]. Chinese painting has many painting techniques, and lines are an important basis for these techniques.

Ink painting has always been a favorite type of painting by Chinese literati, and its ink charm has strong Chinese characteristics [18]. The key of ink painting is the harmony of water and ink. Water is the charm of ink painting. The flowing beauty of water combined with the deep ink color is like black and white dancing on paper, and light and shadow intersect. Ink is generally divided into five colors, which are burnt, thick, dark, light, and clear.

Composition generally refers to the arrangement of objects on the screen, reasonable allocation of space, and good screen management is an important basis for painting [19]. Most of the Chinese paintings refer to the structure of the article, which emphasizes the difference between the real and the false, with a clear distinction between primary and secondary. Although composition is a wild art creation process, it is not ruleless. Many long-term developments in Chinese painting have established many compositional techniques that conform to Chinese aesthetics. Such as full composition, “Zi” composition, corner composition and so on.

3 Feature Extraction

This paper improves the method in Gu [15], extracts the style features of Chinese painting. Then adds these features to the feature matrix.

In order to extract features that can effectively reflect the artistic style of images, This chapter refers to a large of art research literature and some articles on feature extraction and classification of Chinese painting [20, 21]. According to the description in the second section, the general artistic style is expressed in line drawing, frequency, and texture features. Therefore, this article chooses these three characteristics as the artistic style description of Chinese painting. The main methods are:

3.1 Line Feature Extraction

Lines are the most basic component of traditional Chinese painting. Painters use lines to extract, generalize, and abstract natural things and natural scenes. Represent a three-dimensional space with a two-dimensional plane. This experiment uses curvature to represent the fluidity \( F_{line} \) of the line, as shown in Eq. 1:

$$ F_{line} = \frac{{\left( {1 + f_{x}^{2} } \right)f_{yy} + \left( {1 + f_{y}^{2} } \right)f_{xx} - 2f_{x} f_{y} f_{xy} }}{{\left( {1 + f_{x}^{2} + f_{y}^{2} } \right)^{3/2} }} $$
(1)

where x and y represent the coordinates of a pixel in the image, and f(x, y) is the gray value of the pixel. \( f_{x} \text{,}\,\,f_{y} \text{,}\,\,f_{xy} \text{,}\,\,f_{xx} \text{,}\,\,f_{yy} \) are the first, second, and mixed partial derivatives of f(x, y), respectively, and \( F_{line} \) is the Gaussian curvature of the pixel.

3.2 Texture Feature Extraction

Texture feature is a visual phenomenon reflected image, which reflects the surface structure of tissue having a slowly varying or periodic change of the flow properties of the surface of the object. This experiment uses LBP shown in Eq. 2:

$$ {\text{LBP}}\left( {{\text{x}}_{\text{c}} ,{\text{y}}_{\text{c}} } \right) = \sum\nolimits_{{{\text{p}} = 1}}^{8} {{\text{s}}\left( {{\text{I}}\left( {\text{p}} \right) - {\text{I}}\left( {\text{c}} \right)} \right) *2^{\text{p}} } $$
(2)

where p represents the p-th pixel point other than the central pixel point in the 3 × 3 window; I(c) represents the gray value of the central pixel point, and I(p) represents the gray value of the p-th pixel point in the field; s(x) is shown in Eq. 3:

$$ {\text{s}}\left( {\text{x}} \right) = \left\{ {\begin{array}{*{20}l} {1,\,\, x \ge 0} \hfill \\ {0,\,\,otherwise} \hfill \\ \end{array} } \right. $$
(3)

3.3 Frequency Feature Extraction

The frequency of an image is an indicator of the intensity of grayscale changes in the image, and is the gradient of grayscale in plane space. Different frequency information has different functions in the image structure. The main component of the image is low-frequency information, which forms the basic gray level of the image, and has little effect on the structure of the image. The intermediate frequency information determines the basic structure of the image, which forms the main edge structure of the image. The edges and details are further enhancement of the image content on the IF information. This experiment uses Fourier transform to extract the frequency map.

3.4 Add Feature Constraint Algorithm

The three features mentioned above are combined into three channels, and a new feature with a certain width and height is added to the feature matrix extracted by the vgg19 network in the original method to form a constant width and height. New feature matrix. Can be tested in different vgg19 network layers.

4 Experiment

4.1 Build Environment

The experiment uses the caffe framework to build the network. caffe is a deep learning framework that combines expressiveness, speed, and modular thinking. Large-scale industrial applications in academic research projects, start-up prototypes and even vision, speech and multimedia. The experimental hardware platform is Intel Core i7-7800K CPU, 8G memory, NVIDIA GeForce RTX2070 GPU.

In order to verify the validity of the method, we selected some natural pictures and some Chinese paintings on the Internet as experimental materials. The subjective evaluation is used as the evaluation criterion in the experiment.

4.2 Experimental Process and Results

Experiment 1.

Choose a spectacular architectural picture as the content picture and a Chinese ink painting as the style picture. Through a large number of experiments, we found that the effect of adding constraint features to the third layer of the vgg19 network is better than other layers. Therefore, this experiment only operates on the third layer of the network. Figure 1 shows the content image, style image, the composite image generated by the original method, and the composite image generated by our method. Comparing the experimental results, the results of the original method have a great impact on the content, especially the background, and the image has a serious distortion phenomenon. Ours method optimizes these defects. However, the problems existing in the original method cannot be completely eliminated. It is speculated that the style transfer based on the non-parametric method cannot effectively generate a transition for a content map with a large difference between the subject and background colors.

Fig. 1.
figure 1

Synthesis results of architectural drawings and Chinese ink painting

Full size image

Experiment 2.

Choose a nature photo as the content image, and several famous Chinese paintings as the style images. After experimental analysis, it is still the best choice for the third layer operation of the network. The experimental results show that ours method also has certain improvement effects. In Fig. 2, a nature photo as content, and the style image is one of the top ten Mandarin Chinese For the Thousand Miles of Rivers and Mountains, the result generated by this method is more coordinated overall, and the details are also better.

Fig. 2.
figure 2

Synthetic results of nature photos and Thousand Miles of Rivers and Mountains

Full size image

In addition, since the evaluation criteria of the synthesized results are subjective evaluations, 10 different people were found to conduct subjective evaluations on the result shown in Fig. 3 in this experiment. The evaluation results are shown in Table 1. It can be seen that the method in this paper has certain optimization effect on the synthesis result of Chinese painting as a style image.

Fig. 3.
figure 3

Synthetic results of nature photos and famous Chinese paintings

Full size image
Table 1. 10 different people praise the synthesis result (Number of people who think the method in this paper is better/Total number)
Full size table

5 Conclusion

This paper shows a style transfer method for traditional Chinese painting. By improving previous experimental schemes, unique style features in traditional Chinese painting were extracted and added to the feature matrix extracted by the vgg network to achieve the effect of increasing constraints. In the end, a good migration effect was obtained, but it was still unable to achieve the effect of false and real, making people feel that they should draw from the hands of celebrities, and has a certain degree of adaptability to the input image. The main reason is that the paintings in Chinese painting are extremely random and have no regular texture. At the same time, because the evaluation of artistic style images is more subjective, it cannot be objectively evaluated like photographic works. Therefore, we should focus on processing input images and classifying the input images to find the most appropriate content image and style image to match in future. And establish an objective evaluation system for Chinese painting images.