Keywords

1 Introduction

Development of research in sketching and mental imagery in tandem, particularly in design studies for the last ten years can be considered sparse. In 2009, Christensen and Schunn posited that mentally modified and transformed objects serve as both help and hindrance between the perceived world and the imagined world [1]. After a series of seminal studies in expert architects [2,3,4,5], in 2008, Bilda and Gero’s studies were rounded with an exploration that expert architects are able to develop ideas using imagery only. Previously, Athavankar argued that imagery can potentially be substituted for sketching [6, 7]. However, the question remains, what is the vital aspect designers do not have access to if they are not allowed to sketch when generating ideas?

This paper aims at gaining insights about differences of the two types of behaviour by conducting four design sessions of two novice designers and will be closely examined using a theory of visuo-spatial mental imagery. Limitations of the small sample study are foreseen without an attempt to make generalisations between expert and novice designers, but instead novices are used as a starting point to reveal potential when they have less design experience. It might entail less exposure of previously developed design solutions/events/design precedents; or smaller ‘virtual images bank’. Previous findings will be mentioned, major theories of mental imagery will then be posited briefly, and how the preliminary study was conducted will be illustrated. Lastly preliminary results will be discussed.

2 Sketches and Mental Imagery

A summary of findings in relation with sketches and mental imagery in design studies is presented in Table 1. Due to the scope of investigation, the intertwining aspects between idea sketches (hand drawn) and mental imagery in early design stage; this study will focus on points number 1, 6 and 8.

Table 1 Subset of findings of previous studies in external and internal representations
Full size table

In terms of the off-loading roles of sketches; if an operation cannot be handled using imagery only, producing external representations help to reduce the uncertainty. Although they are not the only external representation options, in this paper, the scope is limited to idea sketches only. Bilda and Gero suggest that limited visuo-spatial working memory capacity also affects designers’ ability to retain mental images; therefore they need to be off-loaded [3]. Another plausible explanation is that designers decide that the operation is easier to do externally [8], which then suggests that given the choice, designers consciously make decisions about which mode is more effective.

Table 2 shows that physical elements of sketches underlie these differences. The ability to read-off turns into opportunities to engage with ambiguity, inspectable property of sketches, and the ability to perceive or interpret what has been drawn into creative discovery, facilitate lateral transformation and at the same time archiving new ideas. On the other end of the spectrum, mental imagery also provides better assistance to the idea generation process. For instance, the ease and speed of mental imagery facilitates rapid discovery. The chance to perform virtual design studio inside designers’ head [16] also aided with volatile high resolution images [17] and walkthroughs [16] afford more simulation possibilities. In terms of dealing with absent objects and uncertainty, mental imagery enables retrieval of features and details which are not intentionally committed to memory [18].

Table 2 Differences of sketches and mental imagery
Full size table

Apart from the physical properties of depictions, what properties do sketches hold which are absent in using mental images only? What kind of feedback is missing if there is no conversational dialogue between internal and external representations? These questions will be explored by looking at a new theory of visuo-spatial mental imagery developed by Sima [28,29,30].

The imagery debate often refers to debate between proponents of the first two theories for the last forty years. The debate focuses on two means of representation, depictive and propositional representations. One key property of the depictive format, the Pictorial Theory [31], uses space in a representational medium to represent space in the world [32]. The second theory, the Descriptive Theory [33] rejects that underlying representations of mental imagery are purely propositional and humans use their tacit knowledge to simulate what it would be like to see something during tasks.

The third theory, the Enactive Theory [34] is not as comprehensively reviewed in literature in comparison to the others. Described by Thomas, it focuses on visual perception and visuo-spatial mental imagery. The theory argues that we have sets of inspection processes, seeing and imagining concepts, commonly called schemata. The fourth theory, PIT (Perceptual Instantiation Theory) is built upon the Enactive Theory. Taking the same understanding of visual perception, it is assumed that perception consists of several different specialised types of perceptual information which is selectively used in the process of retrieving information [30]. The formal framework is illustrated in Fig. 1.

Fig. 1
figure 1

Formal framework of perceptual instantiation theory. Source Sima [30]

Full size image

The process of generating a mental image starts with retrieving mental concepts from C-LTM (Conceptual Long Term Memory), these mental concepts than are instantiated together with perceptual information through select-execute-identify processes [30]. Subsequently an interpretation of all identified mental concepts constitutes the mental image of the scene (ibid). This framework will be elaborated in terms of comparison between generating mental images with sketches and without sketches. It is hypothesised when the use of external representations is possible; during the interpretation phase where the most plausible subset is selected, it creates an additional conversational dialogue.

Linkography [35] is utilised to illustrate and test the hypothesis. Developed by Goldschmidt, it is a notation system to analyse design moves and links among them. The graph is a directed graph, however arrows are not used due to the emphasis on the arrangements of the links in a network [35]. Design moves are presented as vertices in network studies, with links between moves depicted by edges. A Linkograph provides an access to the designer’s thinking process at a certain point in time.

3 Methodology

Data was collected from two final year undergraduate design students of an overseas campus of a UK university with two different design backgrounds, architecture and product design. However, comparison between the two domains will not be explored in this paper. Sessions were conducted in English, consent was taken and research ethics approval was obtained prior to recruiting participants and contribution was voluntary.

Each design session lasted 45 min, and each participant did two sessions (mental imagery session-henceforth MI, and sketching session-henceforth SK). Differentiation of these two sessions is adapted from Bilda’s study [23]. The time gap between the first and second session was one month to avoid fixations. Two participants are described as P1 and P2. There are four design sessions in total. Design tasks were: (1) FMS (Flexible Meeting Space) to design a convertible space for a creative industry company in an open plan office, less than 100 m2 footprint and (2) HFS (Hybrid Furniture System), to design a hybrid system of sitting space and dining set-up for adults and toddlers with maximum footprint of 3 m × 3m.

Three to four audio/videos (front, top, side and a replayed pencast from a smartpen, capturing sketching process) in each session were used to compliment the segmentation process. Think-aloud method was used during design sessions and verbal data was transcribed, segmented, encoded based on intentions and was presented in the form of linkographs. The premise is that, if there is a link between two design moves, a line is drawn between the two resembling an edge in networks studies (Table 3).

Table 3 Detailed design sessions for each participant
Full size table

4 Results

Figure 2 shows comparison of four linkographs of four design sessions. In MI sessions, more distributed design moves can be discerned in comparison with the SK sessions in which chunks (graphically distinct triangles) are more prominent. Particularly in the SK session of P2, these overlapped chunks dominate the linkograph. During Ext (externalisation) in the MI sessions, P1 and P2 recalled parts of mental images during a BF (blindfolded) session to draw, which can be seen from links generated for the BF session. During the Ext session, P1 did not refer to the first ideation phase (before move #40); but P2 recalled ideas from as early as move #3.

Fig. 2
figure 2

Linkographs of four sessions

Full size image

Numbers of generated moves in two environments echoed the notion that mental imagery has advantages in terms of ease and speed, and minimum effort compared to physical synthesis (refer to Table 4). It can be assumed that the greater the number, the more design activities occurred. P1 produced 25% more utterances in the MI session compared to the SK session, whilst P2 produced 46% more. Higher link index is a fast indication of higher linking activity and leads to insights into designer’s efforts to achieve synthesis [35]. A link index is the ratio between numbers of links and utterances. Although it is not necessarily a hallmark of creative or good design, in general, link index value of P1 is contrary with P2. P1’s link index in the MI session is higher than SK session, in contrary with P2’s. Threshold of critical moves (CM) is not defined and Table 4 shows the maximum number of links to CM and they are all due to forelinks. Designers shift between divergent (ideation) and convergent (evaluation) modes of thought [35]. It might suggest that more links occur when designers used divergent thought indicated by the forelinks, when P1 and P2 generated ideas. In comparison between MI and SK’s maximum number of links of CM of P1 and P2, in terms of idea generation, mental imagery is more effective than sketches. In terms of idea evaluation, this is a separate research question.

Table 4 Overall comparison of four sessions
Full size table

With regards to posited questions and hypotheses, a partially zoomed in comparison between mental imagery and sketches are extracted. Excerpt from P1’s sessions are used to illustrate. From left to right of the illustration in Fig. 3: partial linkograph-segmented transcription-mapped PIT framework (Fig. 1) are labelled according to segment number. Blue colour font in the transcription records prominent gestures.

Fig. 3
figure 3

P1’s blindfolded of MI (mental imagery) session excerpt

Full size image

The first part (Fig. 3) lasted three minutes towards the end of BF (blindfolded) session. In the middle of generating an idea, P1 retrieved a memory of a previously encountered event (#175) about ‘flexible tables’ which can be put away as walls. The first circle on the right shows how the idea of flexible walls went through the process of select-execute-identify, rationaled (suggesting the use of ‘track on the floor’) and a final decision to be used in the next circle (#183).

The three minute SK session excerpt in Fig. 4 illustrates idea generation using sketches. There were a few instances where P1 did not engage in activities, only verbalising her thoughts (#54, #55, #56 or #63-hovering pen over paper) or just body movement (#50-looked away to imagine how big one meter is). In addition, she was engaged in drawing actions by tracing previously drawn lines (#51, #54, #58, #59 and #61).

Fig. 4
figure 4

P1’s SK (sketching) session excerpt

Full size image

5 Discussion

Mental imagery provides better assistance of rapid discovery in terms of ease and speed as can be seen from Fig. 3. Due to its lower effort when inspecting ideas (more economical in terms of the absence of externalisation), resulting in a higher number of segments in comparison with SK sessions, for both P1 and P2. This does not necessarily suggest a better design outcome (which is not in the scope of this paper). Another explanation is that in the SK session, with the act of sketching and thinking out loud concurrently, due to cognitive loads, participants might generate less verbal data. It can be illustrated in Fig. 4 where a single verbal segmented data may consist of more than one design moves (#51, #58, #60–#63) especially when gesture is involved. As mentioned in the previous section in Table 4, P2 MI session’s link index is lower than the SK session. In contradiction with a previous study of expert architects where there was little deviation, BF link index values are higher. Early indication might suggest that novice designers do not have the same high capability when using mental imagery only. Due novice designers’ lower design experience and smaller ‘virtual images bank’ derived from previous design experience and encountered events, the retrieval process might be different or limited. As far as the chance to deal with incomplete knowledge, transcript #175–#177 illustrate this, and without the danger of not being able to finish a part of sketches, partial incomplete information was used to inspect an idea. It is also noted that a prominent gesture (#173 and #183) might mark the beginning of a chunk of idea, although this needs to be studied in the larger context of the Linkograph and not in isolation.

During the sketching session (Fig. 4), due to the exposure to sketches and surroundings, more comprehensive activities are mapped. Rich dialogue between sketches and mental imagery is illustrated with the use of (partial) sketches as feedback to the MI process. During this kind of ideation process, hybrid mental imagery and drawing process occur. At the beginning of this excerpt, P1 looked away to imagine a distance of one meter. By doing this, P1 recalled a mental picture of one meter from what she saw in the room. The cycle of select-execute-identify is more identifiable, with the use of sketches and gesture. For instance, segment #51 which is comprised of 51a (“one meters..”), 51b (“but this…”) and 51c (tracing); P1 selected the operation to explore about one meter and started to draw the division on the sketch and by tracing she identified the mental concept she then used in her next utterance. A PH (physical) object is added into the map suggesting that at the end of cycle, there are physical properties to be carried for the next cycle. Sometimes it is not the end product of one cycle (for instance, sketch in #54), in this case there was no drawing action in #55, or at the end of cycle (#62) when P1 annotated ‘individual’ space. The hypothesis of interpreting action in the PIT Framework, the use of sketches provides an additional dialogue in a way with available visual properties of sketches help to simulate important parts of design to be imagined. In all instances, the process started after a long pause. It is also noticed that the act of ‘hovering pen over paper’ and ‘staring at drawing’ occur during this process. In addition, with regards to the linkograph, the stage when interpretation happened, was when the idea links occurred.

6 Conclusion

The study has achieved considerations that the use of sketches and mental imagery, both are pertinent. For expert designers in previous studies, one can substitute another. Although mental imagery is not a commonly used in isolation to generate ideas, it can be highlighted that it can be used in design practice as a way to develop ideas. However, novice designers might utilise them differently. Reiterating the questions: (1) What properties do sketches hold which are absent in the use of mental imagery only? (2) What kind of feedback is missing? The physical properties of sketches are inarguably at the core of this. Preliminary study suggests that the availability of visual cues (not only from the drawing) but also the physical context of the designers might also assist the designing process. It is also found that for novice designers, the extent to which mental imagery only can be used as an effective design tool is limited. Mental imagery is an effective aid to generate ideas, however in terms of evaluating ideas it still remains unexplored. The hypothesis is that in the interpretation stage, the use of sketches creates an additional dialogue between parts to be selected. This gives a preliminary answer to the second question. It is also noticed that the role of pauses and gesture in mental imagery sessions, and interestingly in sketching sessions, is plausible. Perhaps in design education, the awareness of benefits and potentials of each way of designing should be introduced in early design education. In terms of digital design tools, the study is hoped to contribute to creation of more intuitive tools based on the interplay, or perhaps mental imagery in isolation which has been underestimated.