Keywords

1 Introduction

Conceptual design is an essential stage of the product development process, where sketch plays an essential role to express design ideas. Before and during the sketch design process, designers usually need to browse lots of reference pictures, and some of the most-related pictures need to be reviewed over and over again. The lookup procedure can cost a substantial amount of time and effort, thus hindering the rapid iteration of product design processes. Moreover, it is difficult to identify and make use of local design elements of reference pictures such as lines. To solve the above problem, this paper tries to design a sketch assistant system based on the existing sketch retrieval technology. The system allows designers to get the existing products with only a few strokes on the sketch screen. In this way, the images of the existing products will appear on the interface of the design software, enabling designers to carry out iterative design with reference to these images.

The problems regarding picture retrieval of sketch have been under examination since 1990s. The purpose of Sketch based Image Retrieval is to retrieve the pictures relevant to the sketch provided by users [1]. However, in terms of the function of sketch retrieval, there is still plenty of scope for improvement.

We distill these synergistic insights into sketch-aider, a system that judiciously combines sketch retrieval technology and product conceptual design. A user study is carried out with the aim of evaluating our system under the assistance of professional designers. The hypothesized advantages of our hybrid workflow for conceptual design are verified.

In the remainder of the paper, Sect. 2 reviews Sketch-based Image Retrieval and related technologies. Section 3 elaborates on the motivation of developing the system. Section 4 explains the EasySketchDesign system and the related retrieval calculation, and introduces the user interface to achieve iterative sketch and refinement process. Section 5 presents a user study to demonstrate its effectiveness in terms of convenience and swiftness. We then discuss in Sect. 6 and conclude in Sect. 7, and then describe future work and limitations in Sect. 8.

A central idea of our work is the use of sketch retrieval to assist designers in conceptual design. This paper proposes a simpler and more practical auxiliary hand drawing interface. When the designer is doing hand drawing, it selects the existing product which bears most resemblance to the current sketch from the library by analyzing incomplete curves and presents it to the designer, which brings lots of conveniences. The design mode can not only save designers repetitive and unimaginative work, but also enhance the design efficiency. Moreover, the neural network based retrieval method we use is accurate in matching and can resist interference. The EasySketchDesign, in particular, can be used to detect conceptual designs which bear a strong resemblance to the existing products.

2 Related Work

Image retrieval technology based on sketch is a kind of technology used for retrieving image from database. This technology is a popular and effective method. Images can be retrieved from database by inputting rough sketches. Sketch-based image retrieval technology is applied to shapes identification, biometric security and other fields. Even if users are not skilled artists, their rough sketch will be sufficient enough to retrieve relevant images from the database [2]. A lot of research has been done to explore SBIR-related technology. Pedro Sousa et al. presented a new way to describe the spatial arrangements of visual elements by integrating spatial proximity in the topology graph, through the use of adjacency weights [3]. A. Del Bimbo et al. studied Elastic match, a reliable method for image retrieval based on contour-based techniques. In this way, similarities in shape between objects in the image and the sketch provided by the user are easily detected [4]. Besides, Contour-based techniques include grid and interest points, such as edge points [5].

In recent years, researchers have gradually applied machine learning algorithms to SBIR. Yang’s article proposed a deep neural network architecture based on Alexnet for sketch recognition and discusses the differences in design between sketch-DNN and DNNs for photographic images [6]. Qi et al. introduced a novel convolutional neural network based on Siamese network for SBIR to gauge the compatibility between image edge and sketch [7]. Bui studied several triplet CNN architectures for measuring the similarity between sketches and photographs, exploring the ability of networks to generalize across different object classes from limited training data [8]. Wang et al. developed an efficient online triplet sampling method to learn the model with distributed asynchronized stochastic gradient, enabling us to learn deep ranking models from large amount of training data [9].

The traditional machine learning algorithm and deep learning algorithm are adequate for such tasks. However, we need large amount of data. In order to identify the required pictures through some hand drawn curves with a relatively small volume of data, we refer to a relatively novel machine learning algorithm called Recursive Cortical Network(RCN) [10]. We then improve the algorithm to complete the task. The algorithm models the contour and the surface of an object separately because the contour and the surface are represented by two independent and interacting mechanisms in human brains [11,12,13]. Therefore, the object is recognized after its contour and surface are identified separately and then combined [14]. In the process of recognition, Loopy Belief Propagation [15] in probability graph model is used to process the extracted contour and the object image to get the result.

3 Motivation: Interactive Sketch Retrieval for Product Conceptual Design

EasySketchDesign is motivated by ideas from multiple and complementary disciplines—the wealth of sketching practice and SBIR.

3.1 Interaction Between Sketch Retrieval and Product Conceptual Design

At present, sketch retrieval is only applicable to image retrieval. In terms of the function of sketch retrieval, there is still plenty of scope for improvement. In the process of product conceptual design, a very simple model is used to find reference pictures. It helps designers complete some tasks and builds a bridge between traditional industrial design and machine learning.

3.2 Auxiliary Product Conceptual Design

A central idea of our work is the use of sketch retrieval to assist designers in conceptual design. This paper proposes a more simple and practical auxiliary hand drawing algorithm. When the designer is employed in making line drawing, the algorithm selects the most similar existing products from the database through analyzing the incomplete curves drawn by the designer, which brings convenience to the subsequent iterative design. This algorithm is highly accurate in matching and resistant to interference. It not only saves designers of the repetitive and unimaginative work, but also improves design efficiency. In particular, the EasySketchDesign proposed in this paper can also be used to test whether the conceptual design is innovative.

4 System Overview and Setup

4.1 Workflow

In this section, we propose an innovative conceptual design workflow that combines sketch retrieval and conceptual design. The sketch image retrieval algorithm used in the EasySketchDesign consists of five main steps as follows.

Establishment of the Image Library.

Users collect pictures to create a reference image library to browse before and during the design process. These pictures can be obtained from the web, or their previous design work.

Outline Detection.

We use OpenCV to process images. All the images in the library are first converted to grey images, and then the Sobel operator is used to extract outlines.

Feature Extraction of the Data Set.

AGabor filter is used to extract the features of images in the library from its outlines. To simplify the procedure, advanced features are extracted and saved for future use. This is the learning process of the model, which is similar to the max pooling in image processing.

Backtracking of Advanced Features and Comparison with Test Features.

In this stage, the advanced features are regenerated through a process similar to backtracking. At the same time, Loopy Belief Propagation is used to compare the generated image and the test data. Choose the image that best conforms to the inputted sketch image.

Output of Search Results.

When the user sets the number of images to be returned, the system will display a set of retrieval results sorted by similarity from high to low.

4.2 Retrieval Algorithm

The overall system diagram of the retrieval algorithm is shown in Fig. 1.

Fig. 1.
figure 1

System diagram of the retrieval algorithm

Full size image

Step 1.

Before the experiment, you need to establish the image library. After designers draw the curves, we need to help them find the existing products most similar to their image. The curves drawn by designers is used as the training image. Specific pictures are collected as test image.

Step 2.

Preprocess the collected data and conduct feature extraction. Since the training image in the experiment is hand-drawn curves and the test image is pictures of real objects, their preprocessing process will be different. For the training data, only the feature extraction is needed before the experiment. For the test data, the traditional image processing technology is needed to extract the outline of the real image. Then the feature extraction should be under way. The first step of feature extraction is as follows.

$$ Y\, = \,X\, * \,w $$
(1)

W is a set of convolution kernel used to extract features. X is the image whose feature should be extracted. Y is a set of results after convolution. \( * \) represents convolution operation. Then the convolution result is processed by non maximum suppression and binarized to ±1 to get the final result Z, which is called the extracted feature. At the same time, the dimension of feature is equal to that of convolution kernel.

Step 3.

Use the improved RCN model to match the preprocessed curves with the preprocessed object image. Learn the features of the curves after processing. The learning process is as follows:

$$ S\, = \,\left\{ {(r,\,c)} \right.|Z\{ r\, - \,a:r\, + \,a\, + \,1,\,c\, - \,a:c\, + \,a\, + \,1\} \, = \,\left. { - 1} \right\} $$
(2)

r and c respectively represent the abscissa and ordinate of any number equal to +1 in Z. a is the suppression radius. S is the result of learning. We can see that the learning process is essentially a pooling process. By suppressing the area near +1 (the square area with a side length of 2a) and keeping the value of the middle point position, the whole image can be learned. Then, select a small number of advanced features. The feature of a point represents the feature of an area. After the learning process, the results are backtracked on the test data set. In this way, the training image is regenerated, and different scores are given for different test images. The meaning of scores is as follows.

$$ D = p(X^{\text{'}} |S^{\text{'}} ) $$
(3)

\( S^{\text{'}} \) represents the image obtained by de pooling. \( X^{\text{'}} \) represents the test image. D represents the score. The formula above indicates that the test image with the highest conditional probability is the best matching image under the condition of retracing the trained image.

Step 4.

Return the best matching results to the designer, analyze the results and complete the experiment.

4.3 User Interface

This paper fully exploits the advantages of Sketch-based Image Retrieval and product conceptual design to produce EasySketchDesign. It expands the application of Sketch-based Image Retrieval and assists designers in product conceptual design. The main interface of EasySketchDesign consists of four parts, including system operation, drawing board, drawing tools and result display, as shown in Fig. 2. The primary function of system operation is to control the operation process of the whole system, including screenshots, search, update, etc. The sketchpad is shaped like a rectangle. Users can draw directly on the sketchpad. The result display demonstrates the retrieval results of the system. The image list is arranged in descending order according to the similarity with the sketch. Figure 3 shows the overall workflow of our system.

Fig. 2.
figure 2

User interface. (a) System operation. (b) Drawing tool. (c) Drawing board. (d) Result display.

Full size image
Fig. 3.
figure 3

System workflow. (a) Drawing. (b) Capture. (c) Update. (d) Modify figure.

Full size image

4.4 Implementation

The user interface of EasySketchDesign is developed using C++ with Qt creator. We use a Wacom digital display with a stylus as input device. During the sketch design process, the user uses a screen snap tool integrated into the sketch interface to generate a sketch image. Next, the captured sketch image is automatically inputted to the background image retrieval procedure, which is developed using Python. Then the retrieval procedure compares the features with the inputted sketch image and output the similar images in the reference window of the Qt interface.

5 User Study

We evaluated our system with professional illustrators and designers. Our study aims to exam the core functionality of the system, and to find out its limitations. We also include a purely sketching condition to compare sketching with EasySketchDesign.

5.1 Participants

We recruited eight participants (age 24–32, 4 female) for our study. Participants were all experienced with sketching and Hand-drawn tools. Participants were compensated for their time.

5.2 Procedure

The study was conducted in our lab, which lasted a total of 50 min with three phases.

Introduction and Training.

Participants were given a general introduction that explained the entire process and given time to get used to the EasySketchDesign. Participants practiced using the EasySketchDesign until they felt confident. This phase lasted a total of 10 min.

Fixed Tasks.

We designed fixed tasks to compare sketch with simple sketch. In order to eliminate learning effects, we balance the order of systems among participants. This phase lasted a total of 15 min.

The task involved drawing the left view of the car. There were two groups of participants. One group drew with EasySketchDesign. The other group drew in a normal way. In the process of drawing without EasySketchDesign, participants can refer to pictures in the local computer folder. Figure 4 shows the sketching of two participants (participant number P3 and P5).

Fig. 4.
figure 4

(a) Sketch drawn by P3 with EasySketchDesign. (b) Sketch drawn by P5 without EasySketchDesign.

Full size image

Freeform Design.

Finally, in the third phase, participants were enabled to design whatever they wanted in the system. After the study, they needed to finish a questionnaire. This phase lasted a total of 25 min.

5.3 Results

Results for Fixed Tasks.

For the fixed tasks, we used completion time to conduct quantitative measure. Then, we analyzed the time needed in the two different situations mentioned above.

The result shows that it took participants less time to complete the tasks when using EasySketchDesign. The completion time of each participant is shown in Appendix A. The average time of producing sketches by purely sketching (m = 623, σ = 38) was longer than that of producing sketches with EasySketchDesign (m = 535, σ = 30).

However, we should take these statistics into careful consideration, because our concentration was on qualitative measures, and the tasks were not carefully controlled. The study shows that two primary factors are behind such results. First, adapting to the EasySketchDesign costs more time and required more mental effort. However, so long as participants master the EasySketchDesign, they will draw designs with much ease and more efficiency.

Qualitative Feedback for Freeform Design.

Participants successfully used EasySketchDesign to complete the design task. During the short design period, participants produced the sketch design of helmet, shampoo bottle, as shown in Fig. 5. In general, participants made positive response to EasySketchDesign. From participant 1’s comments, it was clear that the system was “a good way to assist in conceptual design”, but that “it was necessary to become familiar with the use of the system”. Below, we report the partial feedback received for individual components.

Fig. 5.
figure 5

(a) Helmet. (b) Shampoo bottle

Full size image

Provide Picture Reference In real Time.

Participants found this feature useful. They found that the accurate retrieval of the desired images in real time can not only improve the conceptual design but also help identify sketches that highly resemble the existing products. Participant 4 said, “Simply draw one or more lines based on your idea and quickly find an existing product image that fits your idea for reference. It makes me feel very happy.”

Ability to Combine Concept Design and Image Retrieval In real Time.

Tests show that EasySketchDesign can be used to find the desired images. It costs less time compared with participants looking for images in the local computer folder on their own. Participant 7 said, “It’s very convenient. You don’t have to go through a lot of pictures to find the right one. It takes very little time. However, it still took a while to wait for the retrieval results”. “It consumes me less time so that I can spend more time in referring to models.”

Participants also provided us with feedback on other aspects of the system. At present, content-based image retrieval is widely used. Whether it can be included in our system or not, this is an interesting and constructive suggestion for future work. In addition, some participants suggest referring to other kinds of products when drawing sketches. For instance, when drawing sketches of cars, we can not only refer to pictures of cars but also those of trucks and airplanes. Besides, some participants hope they can retrieve 3D models through sketches.

6 Discussion

Our evaluation indicated that our core ideas enjoyed popularity among the participants. They were enabled to use the system to create the designs they wanted. The sketch design completed by the participants shows the effectiveness of EasySketchDesign and creates a new interactive system for conceptual design. However, users’ feedback demonstrates some limitations, which indicates that there is room for development in the future.

7 Conclusion

Through the improvement of a very novel machine learning model, this paper aims to combine product design sketch and sketch retrieval (SBIR), and increase their interaction. An interaction mode and a prototype system are developed, which benefit the design conception, expression and communication. Users’ feedback proves that our system is useful and effective, and can address various design tasks. We hope this work can provide reference for researchers of industrial design and machine learning, and promote the combination of industrial design and machine learning.

8 Limitations and Future Work

There are some differences in users’ cognition. Different users may have different understandings toward the same sketch [1]. These methods are usually combined with sketch retrieval algorithms to enhance the accuracy of retrieval results [16,17,18]. Therefore, we should consider the semantic information contained in the sketch and the image to be retrieved so as to enhance the accuracy of sketch retrieval results.

Further expand the data set to improve the system. The enlargement of the data set enables designers to refer to pictures of different kinds of products, giving them more inspiration. Besides, 3D model can be retrieved through sketches, providing more detailed reference for designers. However, this is a challenging task.

Although the method has achieved some results, it also has some limitations. Human beings usually have a focus of observation in the process of recognizing objects. However, there is no difference between the features in this paper’s experiment. Therefore, in the following research, we can solve this problem by adding weight to different features so as to improve the stability of the model. This is what we need to study in the next stage.