Keywords

1 Introduction

The booming of portable devices and networks helps to the development of innovative interaction pattern and operation behavior in the area of human-computer interaction. The design thinking of man-machine interface is being migrated from the pattern research of graphical interface to user-centric interactive design [1]. For improving the efficiency of knowledge utilization on a user-centric interface, a lack of an available method for knowledge induction, sorting, archiving may make a contribution to the poor reuse of existing data in the process of user interactive design. Since knowledge reuse, with its guidance, foreseeability and reappearance, is instructive for the interface design and interaction theory. In this paper, taking animation cases as application background, we make the most of knowledge reuse to present a sketch animation editing method that supports user with simple, natural sketch drawing on operating and organizing animation appropriate to user’s intention with a low cognitive load. As animations are ubiquitous in modern visual media, the animation editing is an interactive, visual prototype with an interface resembling a sketch editor or drawing application, which provides an open work surface similar to the user’s sketching lines. The user can sketch static illustrations or outline annotations directly on the objective animation using a simple sketch operation. The editing metaphor is extended with tools for sketching, interactive visual elements on the animation surface. Considering that user’s editing stroke represents the timeline of an animation that can be used to achieve any operation like cutting, splicing, selecting or modification, We take advantage of the ability of sketching to provide efficient interaction directly to perform knowledge reuse for user design with the animation elements in order to make it easier and faster to work with animation editing.

The main contributions of this paper include: (1) Analysis the interactive behavior of user-computer to provide a sketch-based user model, which provides available theoretical guidance and assessment principles for the natural sketch editing method. (2) Demonstrate a sketch modeling of the multi-layered solution mechanism to achieve sketch structural recognition and understanding. (3) Provide a sketch-based animation editing platform that supports knowledge reuse for user facilely access to animation design in a natural environment. The work we conducted shows that it enhances the accessibility and individualization during the process of user drawing design and helps to enhance user experience in human computer interaction.

2 Related Work

Animation design is one of the most challenging problem-solving activities that the modern system faces. The cognitive complexity of design problems and the respond-time pressure due to improved interactive efficiency force designers to pursue efficient processes and strategies [2]. Natural editing method have become the central feature of user intention expression and are becoming the more fundamental and critical part in human interaction [3]. The traditional freehand writing on a paper is that simple and provides good user experience during the knowledge understanding. Most current animation designs for interactive interface are intuitive and multi-style which present different, rich menu items for users accessing operations. But they cannot support the interactively creative ideas description and achieve natural interaction well, which is important for user experience in the process of communicating with system. With the popularity of computers and pen interaction technologies, ideas of new methods and interactive technologies are incited to aid the sketch design and application. We prefer combining technologies of sketch-based recognition and understanding to explore an effective knowledge reuse in natural animation editing system. Some related techniques are involved, such as cognitive model, knowledge reuse and sketching technology.

Cognitive Model.

Since the user tasks in the systems are becoming more complex, the benefit of systematic interface design receives more emphasis [4]. A good interface design procedure should elicit the user’s task needs and task knowledge, which to some extent, leads to the demand of user cognitive factor on the special interactive interface [2]. To establish effective design strategies, the practical cognition of different users’ needs to be considered [5]. Exploring cognition mechanism of user behavior helps to illustrate the user cognition model in the calculation process of user analysis, modeling and decision-making [6] when they interact with the interface. To realize natural and efficient interaction, we need to build the mapping mechanism to make computer understand the human cognitive process and shorten the cognitive distance between themselves thus reducing the burden of users’ cognition. In this paper, we explore the user behavior and characteristics in the cognitive process to provide a sketch user model.

Knowledge Reuse.

Natural interface design is becoming a more complex task due to the increasing needs of functionality and convenience of animation editing systems. Since User’s constrained cognitive capability has made natural interface challenge further, we should optimize the interaction between the user and interface considering user goals, cognitive capabilities and available interface intention in the process of animation editing. Thus the concept of knowledge reuse has been proposed to minimize many redundant works and save the time cost for user editing and drawing [7, 8]. Among the existing approaches to achieve systematic knowledge reuse, some may appropriately focus on the preservation of design rationale for the issue-based information systems [9,10,11], some may consider the component-based software development with reusable modularized components practiced to depict object-oriented modelling techniques [12, 13], and others may prefer case-based reasoning techniques to find a prior problem similar to the current problem and provide a modified solution [14, 15]. This techniques have been applied to various design domains such as architecture, mechanical or physical devices and software [16,17,18]. In this paper, we probe the opportunity to achieve better user design by presenting relevant design cases based on sketch-related method. The sketch knowledge representation is therefore taken as the most important success factor, for which we propose organization of cases for users efficiently access to the animation design.

Sketching Interface.

Sketch-based interaction presents a new generation natural user experience that breaks the bottleneck of current WIMP pattern and adopt the nature and efficiency of pencil and paper based interaction. Freehand sketching is a natural and crucial part of everyday human interaction, especially in early design processing [5]. Sketching technique mimics the tradition pen and paper that represents a natural way of thinking about ideas, its interface satisfies users in a freeform and flexible manner, allowing them to use their pen to draw sketches actively when interacting with computer systems [19]. With sketch interaction and recognition method, we provide user feedbacks to improve and enhance the learning process. By creating sketch-based interface, users can editing data more easily while keeping a natural style of interaction [20, 21]. Sketching technology has become popular for interactively design in different domains, like animation [22, 23], garment design, mathematics sketching, botanical modeling [24], floral modeling [25], hair style modeling [26, 27] and so on. In these systems, sketching can be used to create the prototype of design quickly. However, in our work, recognizing and understanding of sketches user depicting and designing can be performed based on the animation design.

3 Knowledge Reuse for Adaptive Sketching User Interface

3.1 Sketch-Based User Interaction Model

User model aims to explore the user characteristics in the process of the interaction by grasping user’s cognitive habits, predicting their behaviors, thus to provide theoretical guidance for reuse design of user interface. User plays a central role in generating knowledge information from the snippets of data emerging from visual interaction, which enables users to analyze huge information spaces in order to support complex decision making and data exploration. Exploring the process of knowledge generation based on user’s cognition helps to effective modeling of user visual analysis and interaction. In this paper, we adhere to the principle of user-centered interactive characteristics and rule of sketch-based editing method, referenced to the knowledge generation process [28] to provide a sketch-oriented user interaction model (see Fig. 1), which illustrates the usual interactive process that enables users to interact with the system using sketch interaction. The model demonstrates the cyclical process of user-computer interaction, the user part illustrates the knowledge generation process of the human, which depicts a reasoning process composed of exploration, verification, and knowledge generation loops. The interaction part presents the interactive information on the sketch interface with its different context provided so that a sketch-oriented object can be developed to support various operations and applications. The contexts contained in the system includes, User context: experienced designers or novice. Environment context: devices, pen locations, pen-styles, colors, symbol, diagram etc. Sensed context: time, sequence, color and positioning information etc. Interpreted context: first level derived context from sense context: classification of texts, symbols, drawings, gestures, etc. Second level derived from first level: geometry, topological relations, and sketch constraints.

Fig. 1.
figure 1

Sketch-based user interaction model

Full size image

3.2 Adaptive Sketch Recognition and Understanding

Due to the various ways user depicting sketches and its intentions, uncertainties over the process of interaction make it appropriate to consider the dissimilarity among users to develop knowledge reuse sketch context and constraint [29]. Thus to achieve an adaptive sketch-based interface for facilitating the design reuse process, modeling the sketching process is of great significance to improve the sketch recognition and understanding in the process of animation editing and interaction. We define the sketch including strokes that involves different constraints, strokes should be operated in the vector format, since a stroke is a set of sequent points captured from an input device during a time period between a pair of pen-down and pen-up events. Each point has a time stamp, with which we can recognize and compute the stroke to understanding the whole sketch structure. Admitting that different users may draw several strokes continuously or overlapping when they are required to finish a straight line, we need to group some strokes into different edges, which is a high level geometrical concept. Any modifications in either sketch strokes may contribute to the corresponding changes in another, which demonstrates the necessity of sketch modeling (Fig. 2 shows the corresponding sketch modeling).

Fig. 2.
figure 2

The sketch modeling for recognition and understanding

Full size image

To reuse the knowledge in interface design that users input the sketch data, system adaptively responds the retrieval and recognition for providing the existing similar recommendations. Adaptive recognition and understanding here means it can spontaneously adjust to variations of any sketches the user editing and provide optimal result realignment. Since the sketches users depicting presents the integrated concepts of their intents, we need to split them into every single meaningful pieces and then support a partial matching for sketch recognition [30, 31]. Figure 3 gives examples of a sketch drawings, structural recognition and understanding. For a set of sketches, detection and segmentation of pieces (nodes) should be first conducted. Considering that close distance between strokes among every piece may lead to inefficiency of recognition by simply computing the distance of strokes, we propose a new method to confirm the relative position of sketch nodes and use RNN classifier to train and recognize the feature vector of that obtained position. On the basic of recognition for connecting lines, we aim to merge the strokes according to the distance of strokes and the density of a stroke set. Denote that \( dist\left( {S_{i} ,S_{j} } \right) \) as the distance between stroke set \( S_{i} \) and \( S_{j} \), \( stroke\_thres \) as the merging threshold, while \( dist\left( {S_{i} ,S_{j} } \right) < stroke\_thres \), taking the merging operation since \( stroke\_thres = \frac{3}{2n}\mathop \sum \limits_{i = 1}^{n} d_{i} \), where n represents the total number of strokes in a sketch, \( d_{i} \) represents the diagonal length of the select box that can contain stroke i. Meanwhile, computing the stroke density \( den(B_{i} ) \) that has been merged and its density threshold \( block\_thres \) using \( den\left( {B_{i} } \right) = \frac{{\mathop \sum \nolimits_{j = 1}^{k} len_{j} }}{{dia_{i} }} \), in which \( len_{j} \) is the length of stroke j, \( j \in B_{i} \), \( dia_{i} \) is the diagonal length of the bounding box of stroke set \( B_{i} \). If \( den\left( {B_{i} } \right) < block\_thres \), abandon the merging. With the iteration of recognition for each piece of strokes, further work should be conducted to achieve the recognition of the whole sketches a user depicting. In Fig. 3, the drawing nodes are shown on the left side, the corresponding recognitions are generated by circling every piece on the right side. Its recognition can be similar to the process of stroke merging to obtain a set of stroke \( B_{i} \). The central point of \( B_{i} \) is denoted \( C_{i} (x_{i} ,y_{i} ) \), and two endpoints of the arrow, \( p_{1} (x_{1} ,y_{1} ) \), \( p_{2} (x_{2} ,y_{2} ) \) constitute the vector \( v_{1} (x_{1} - x_{i} ,y_{1} - y_{i} ) \), \( v_{2} (x_{2} - x_{i} ,y_{2} - y_{i} ) \), in which their angles \( \alpha 1 \) and \( \upalpha2 \) helps to verdict the relative position of each stroke set. If \( cos\alpha 1*cos\alpha 2 < 0 \), we conclude that stroke sets are distributed at the top or bottom side of the arrow. If \( cos\alpha 1*cos\alpha 2 \ge 0 \), they are distributed around the arrow. Among of which,

$$ cos\alpha 1 = \frac{{\left( {x_{1} - x_{i} } \right) \times \left( {x_{2} - x_{1} } \right) + \left( {y_{1} - y_{i} } \right) \times \left( {y_{2} - y_{1} } \right)}}{{\sqrt {\left( {x_{1} - x_{i} } \right)^{2} + \left( {y_{1} - y_{i} } \right)^{2} } \times \sqrt {\left( {x_{2} - x_{1} } \right)^{2} + \left( {y_{2} - y_{1} } \right)^{2} } }} $$
$$ cos\alpha 2 = \frac{{\left( {x_{2} - x_{i} } \right) \times \left( {x_{2} - x_{1} } \right) + \left( {y_{2} - y_{i} } \right) \times \left( {y_{2} - y_{1} } \right)}}{{\sqrt {\left( {x_{2} - x_{i} } \right)^{2} + \left( {y_{2} - y_{i} } \right)^{2} } \times \sqrt {\left( {x_{2} - x_{1} } \right)^{2} + \left( {y_{2} - y_{1} } \right)^{2} } }} $$

After obtaining the relative position of the stroke sets, we can confirm whether they belong to the arrow by judging that distance d ≤ threshold or d > threshold thus to achieve the recognition of each piece of strokes.

Fig. 3.
figure 3

An example of sketch drawings, recognition and understanding

Full size image

3.3 Knowledge Reuse for User Interface Design

The objective of the sketch-based animation editing system is to provide users with benefits of a traditional pen-pencil tool that has taken the effective design knowledge reuse into a consideration. The design of history backtracking develops effective means of adapting to the history information reuse for it helps to amend the sketch editing at any time in history in the process of interactive design. An orderly history backtracking means that stretching the design process back to previous drawing result in the form of backtracking command. For achieving the history knowledge reuse in the sketch interface, we provide a backtracking algorithm of random concept based on user’s design intent as follow.

Denote the current collection of selected objects as \( P = \left\{ {\psi_{1} ,\psi_{2} , \ldots ,\psi_{m} } \right\}, j,k \in (1,n) \), FirstPrev = null, SecondPrev = null, the design node of conceptual object \( \psi_{i} : \)<eleId, attributes, head>. \( \psi_{i} .{\text{head}} \) shows the head pointer of the constraint chain, the constraint node is defined as Cons: <Type, ElefirstID, ElesecondID, Attributes, ConsNextforfirst, ConsNextforsecond>, in which ConsNextforfirst points to the next constraint node of object ElefirstID, and ConsNextforsecond points to the next constraint node of ElesecondID,

  • For step 1, j = k, if j > n, it goes to step 4, i = j + 1, the current design object is \( \psi_{j} \), with the index value, the current object and its mapping constraint chain can be available.

  • For step 2, searching for the constrain chain of current object, if i > n, k = k + 1, then it goes back to step 1, and taking search matching of the two design objects that the constrain nodes had recorded as <\( \psi_{j} , \psi_{i} \)> (<\( \psi_{i,} \psi_{j} \)>). If it succeed, goes to step 3, or i = i + 1, then goes to step 2.

  • For Step 3, the current constrain node is Con, FirstPrev records the previous constrain pointer or the mapping head pointer of \( \psi_{j} \), SecondPrev records the previous constrain pointer or the mapping head pointer of \( \psi_{i} \), FirstPrev = Con.ConsNextforfirst, SecondPrev = Con.ConsNextforsecond, then delete Con, goes to step 2.

  • Step 4, end of the program.

The sketch-based backtracking in the process of drawing design helps to enhance the users’ thinking process, with which users can take advantage of the ability of the reuse of history information to achieve an effective design on sketch editing. The system we provide allows users to input their own sketch knowledge to design and editing the animation, which includes animation importing and modification on the input knowledge by only using sketch interaction. The input knowledge they produce can in turn be added to the knowledge base for further reuse and reedit. If a regular user leaves, the loss will be prevented because his/her entering knowledge and experience will have already been saved for reuse. Figure 4 demonstrates the hierarchy structure of the editing system for knowledge design process, which includes four layers: data layer, computing and processing layer, application layer and interaction layer, in which, data layer depicts the animation knowledge and experience during the inputting process, as well as the original (reuse) input animation the system had provided before. It provides general data access for the external application. Computing and processing layer provides functions to recognize and process the inputting data collected by sketching from the users. As leading to the personalities of the systemic achievement. Application layer provides functions of creation, modification, save of the animation input drawings. And interaction layer provides the flexible sketch-based interaction to improve the natural human interaction with the purpose of reducing the cognitive load of different users. Figure 5 illustrates an example of the sketching editing of the animation and their structural recognition. When designing and editing any animation using sketches, users can save the depicting into the sketchborad for their reusing and editing with the functions toolbar for modification.

Fig. 4.
figure 4

The hierarchy structure of system

Full size image
Fig. 5.
figure 5

Example of sketch editing and processing

Full size image

4 Conclusion and Future Work

A user-centric natural interaction, has the potential to alter the way that user’s usual access to the knowledge creation process with distinct advantages. In this paper we present a natural sketch-based animation editing and design based on knowledge reuse, aiming to improve the design efficiency and user experience since sketch-based interaction helps to achieve the fact that reducing the user’s cognitive load. The research involved detailing and demonstrating a sketch-based user interaction model, a method of sketch modeling and a supporting knowledge reuse user interface. It is capable and applicable for modeling various sketching interfaces and knowledge modeling applications. Of course, various input constraints for sketch structure recognition and understanding can be considered as one kind of improvement for our future work. A long term goal is to explore the algorithms of extending the abilities of adaptation and recognition of complex sketches, enriching them with further user interaction model. These characters would be beneficial to improvements of more natural interactive applications, capable of addressing the limits due to different users’ cognitive ability, capturing users’ intentions and then performing available presentations.