Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Problem Statement and Context

1.1 Benefits of Design Sketching

Engineering students must achieve expertise in a wide range of computing skills during their studies in order to properly prepare them for their future careers. However, design sketching continues to be a valuable skill that is applicable in a variety of use cases, since this skill enables engineers to translate their design thoughts effectively on a visual medium. These use cases can range from proficiently producing rough preliminary sketches and polished drafting designs using pen and paper, to seamlessly interacting with gesture-driven intelligent user interfaces and emerging sketch-based modeling applications with a stylus-and-touchscreen. Furthermore, engineering students can take advantage of the educational benefits associated with sketching in general, such as enhancing the communications of existing ideas [6] and inspiring a generation of new ones [5]. Mastery of design sketching can lead to benefits in creating diagrammatic representations that hold many advantages over textual representations alone, such as grouping information more usefully for improved problem-solving inference [8] and better representing ideas for conveying engineering ideas [1]. Combined with current engineering education, sketching therefore provides assistance in accessing knowledge that is based in the applied sciences through aesthetic and social inquiry.

1.2 Challenges of Traditional Design Sketching Instruction

While mastery of design sketching enables engineering students to exploit this skill for their coursework and subsequent real-world usage in post-university careers, the process of them achieving mastery provides its own challenges. From our interviews with two industrial design professors from a large research university and from university students who are learning how to sketch in order to apply this skill to their coursework, we were given insights into both the traditional sketching pedagogy and the difficulties that both students and instructors experienced. Among these difficulties for learners include low self-efficacy [2], discouragement from comparison with peers, and lack of motivation to practice and self-regulate their learning [11]. As for instructors, factors include difficulty in catering their teaching to each and every individual. Traditional approaches for teaching sketching principles primarily stem from either in-class instruction and dedicated workshops, which are constrained by the time and availability of experienced human instructors; or rote self-practice and sketching guidebooks, which lack supervised feedback for alleviating the possibility of incorrect learning and bad sketching habits [10].

1.3 Prior Work

With the challenges associated with traditional design sketching instruction, there is potential in engineering students taking advantage of the growing ubiquity and accessibility of pen-capable computing devices to utilize intelligent pen-driven educational applications, where such an application can guide the students’ study of design sketching fundamentals through sketching practice and automated human instructor-emulated visualized feedback. We can see such examples for sketching in the domain of art (e.g., [3, 4]), where sketching interfaces guide students in drawing foundational concepts by providing step-by-step cues on drawing and automated real-time feedback on their drawing performance. Similar examples also exist for assessing young children’s developmental progress through sketching (e.g., [7]) using playful interfaces that additionally guide children how to sketch basic shapes. However, there is a lack of work in developing appropriate educational applications that cater specifically to users with developed fine motor skills with little to no design sketching background for more technical domains such as engineering and design.

1.4 Proposed Solution

In this paper, we therefore introduce PerSketchTivity, our intelligent educational application for teaching design sketching to users through sketching practice and corresponding automated feedback on their sketched input, in order to address the various challenges that both students and instructors encounter during design sketching instruction. Our approach for automatic understanding of freehand sketch input relies on: (1) a variety of geometric, feature, temporal, contextual, multi-modal, and domain information that provides context about what shape is being drawn on the screen, (2) a way to describe the sketch information, and (3) the application of machine learning techniques that take advantage of, and learn from, the information provided. Additionally, insights from educational sketching books (e.g., [9]) help influence the design decisions of our interface’s interactive lessons using existing successful practice for sketch instruction.

2 Method Employed

We implemented a web-based intelligent educational application that utilizes Paper.JS—an open source javascript vector graphics scripting framework—and which runs on top of HTML5 Canvas. Our application’s interface is therefore platform-independent and appropriate for pen-driven input modalities. The accompanying hardware setup that we employed for our application consisted of a Wacom Cintiq touchscreen monitor that accepted accurate stylus input, but can easily be adapted to other hardware setups that accept stylus input such as mobile tablets and tabletop touchscreens.

In the latest version of our application, we designed an interface that consisted of several exercises—specifically those for teaching users how to draw straight lines, squares, and circles—while also accommodating future extensions for lessons on more advanced geometric primitives. We elaborate on our application’s different exercises in this section.

Fig. 8.1
figure 1

Interface screenshot prompting the user to practice sketching straight lines at their own pace by connecting dots, and with accompanying feedback based on their sketching performance

Full size image

2.1 Straight Line Exercise

Straight lines are the most fundamental technique of sketching, and the very first sketching technique designers and engineers are taught is typically straight lines, as they form a fundamental skill from which to construct all three-dimensional forms. Therefore, we introduce straight lines as our first interactive lesson within our application. Students are exposed to a “connect the dots” style of exercise, and typically practice repeatedly hundreds of times to master the technique. With traditional pedagogy, there is a focus on accuracy, speed, and line quality, so by leveraging existing sketch recognition approaches, we are able to provide real-time feedback in the form of color-coded bars (Fig. 8.1), as well as an evaluation that provides students with additional metrics regarding their accuracy and speed (Fig. 8.2).

Fig. 8.2
figure 2

Interface screenshot of automated feedback containing metrics regarding their accuracy and speed on sketched straight lines

Full size image
Fig. 8.3
figure 3

Interface screenshot prompting the user to practice sketching squares, with accompanying feedback on their sketching performance

Full size image
Fig. 8.4
figure 4

Interface screenshot prompting the user to practice sketching circles, with accompanying feedback on their sketching performance

Full size image
Fig. 8.5
figure 5

Preliminary interface design for proposed future exercises that build and extend from the existing exercises

Full size image

2.2 Square Exercise

Comprised of straight lines perpendicular to their adjacent neighboring lines and parallel to lines on the other side, we introduce squares as the next basic shape for students to practice learning sketch technique on. Advancing from line exercises thus allows students to sketch multiple straight lines, in order to construct basic two-dimensional forms (Fig. 8.3). Multi-stroke detection allows students to sketch four lines for the squares.

2.3 Circle Exercise

Using the knowledge of how to draw squares, our application subsequently allows students to practice sketching perfect circles inside reference squares (Fig. 8.4). Our application uses scaffolding in this way as a fundamental concept for students to experience during their sketching lesson, as more complex forms must similarly be “constructed” from basic forms.

2.4 Future Exercises

With students completing their lesson practice on these prior sketching fundamentals for two-dimensional geometric primitives, we would like our application in the future to incorporate more advanced exercises that enables students to learn about perspectives and how to sketch three-dimensional geometric primitives such as cubes, cylinders, cones, and sphere (Fig. 8.5).

3 Results and Evaluation

3.1 Usability Study

In order to evaluate the performance of our current intelligent educational application for design sketching instruction, we recruited eight university students from a large research university for a usability study. They were from a variety of disciplinary backgrounds such as industrial design, mechanical engineering, computer science, and human-computer interaction. These students covered a spectrum of self-reported sketching experience from beginners with little to no sketching experience to intermediate-level students who still wanted to improve their ability.

For our usability study, we asked the participants to practice on four developed exercises, two of which utilized sketch recognition approaches. The focus of the exercises were on fundamental sketching components of lines and circles, each with two of their different exercise styles. One exercise style involved sketching many-at-a-time with an evaluation at the end, while the other exercise style involved sketching one-at-a time and evaluating between sketches. Since we designed our interface as a computer-assisted educational application, we therefore focused on learning outcomes and whether or not the pages and exercises were accomplishing the learning goals instead of traditional usability heuristics (e.g., task completion time, error rate). Our evaluation therefore placed greater emphasis on qualitative feedback, but also included collecting quantitative data using performance metrics to measure the effectiveness of pagination, exercises, and the user interface itself for determining their usability effectiveness.

Fig. 8.6
figure 6

Interface screenshot of lesson map page, which shows the user what they will be learning, what they already learned, and what can be navigated in the educational modules

Full size image

3.2 Lesson Map Page Evaluation

One component that we included in our application was a lesson map page for showing the user what they will be learning, what they had already learned, and what can be navigated in the educational modules (Fig. 8.6). Using a self-assessed ten-point scale, our participants gave an average of 9.3 in terms of understanding what they will be learning, and an average of 8.0 for understanding what they had learned. In addition to the accompanying qualitative data, participants overall understood what they were going to learn from the lesson map page, but were less clear on what they had already learned or what they would learn next. Their feedback suggested having a more clear path between lessons and a visual indicator of their progress through each lesson.

3.3 Lesson Overview Page Evaluation

Another component included in our application was a lesson overview page for educating users on a specific skill or technique, as well as serving as a navigation for the exercises (Fig. 8.7). Using a ten-point scale, our participants gave an average of 9.4 in terms of understanding how to sketch the prompted shape using video, and an average of 8.9 for understanding how to sketch the prompted shape using pro tips infographics. In addition to the accompanying qualitative data, the lesson overview page received high performance metrics, but participants largely expressed having a desire for less scrolling and more clicking interaction from this page.

3.4 Shape Exercise Evaluation

The main component of the application was the different shape exercises, and participants were asked to progress through the different sketching lessons for the various prompted shapes. Overall, users on average rated the second iteration of the shape exercises at least 8.6 for initially understanding what to do in the exercise, at least 9.1 for their level of focus during the exercise, and at least 9.0 on their ability to sketch the prompted shapes more accurately. The accompanying qualitative data varied for each shape, but the general consensus was that the shape exercises were well-received while some had issues with the timing aspects of the application, such as the exercises being timed and timing oversensitivity of the prompted cues.

Fig. 8.7
figure 7

Interface screenshot of lesson overview page, which educates the user on a specific skill or technique and serves as a navigation for exercises

Full size image

3.5 Interactive Coach Evaluation

A supplemental component of the application was the incorporation of an interactive coach to warm up users to the educational experience within the application. We proposed three candidate interactive coaches, and participants were most receptive to the candidate whom visually resembled historical artist Leonardo Da Vinci (Fig. 8.8).

Fig. 8.8
figure 8

Visualization graph of candidate interactive coaches and the ratio of preference votes

Full size image
Fig. 8.9
figure 9

Interface screenshot of assessment display

Full size image

3.6 Assessment Display Evaluation

One other component in the application was the assessment display, which was provided after completing a lesson (Fig. 8.9). Overall, participants reacted positively to and feeling accomplished with the automated assessment of their sketches, especially with real-time deviation lines and regardless of minor evaluation errors. One remark from participants was that they were open to assessment being displayed as either a five-star rating or as a full performance metrics overview, which we can implement in the future as a toggle. An additional remark was an option to view a history of their previous scores to self-determine their own improvements of their sketching ability over time.

Fig. 8.10
figure 10

A user during one-at-a-time exercises, where the participants expressed these types of exercises as “better practice” and were able to get in to more of a state of flow

Full size image
Fig. 8.11
figure 11

A user received real-time feedback from deviation lines generated from our sketch recognition approaches, which participants praised for their usefulness

Full size image
Fig. 8.12
figure 12

An example of a user encountering confusion of what to do next after an exercise that is common with participants

Full size image

3.7 Overall Insights

From our evaluation, we observed the usability testing demonstrated the value of both exercise designs in terms of motivation, states of flow, and subjective opinions on their effectiveness as learning tools. We discovered that one effective approach would be to combine the positive aspects of both exercise styles into a single exercise style (Figs. 8.10 and 8.11).

A major flaw in our current interface design that we partially anticipated was that participants did not fully understand what to do before practicing a new exercise (Fig. 8.12). We believe that more care should be taken in guiding users through the experience and without surprising them. Furthermore, we observed that scrolling should generally be eliminated from the interface due to the way people interacted when using styluses.

4 Contributions

This paper introduces our preliminary prototype and evaluation of the PerSketchTivity system. Our results show that traditional paper-based methods for learning sketching can be translated to the digital medium. Further research will evaluate its effectiveness in raising motivation, self-efficacy, self-regulated learning, and spatial reasoning skills.