Studying interaction density in co-design sessions involving spatial augmented reality

Abstract

The participation of end-users in co-design sessions has always been challenging. Despite a long history of trials and research, finding the best way to go about this form of collaboration in early innovation phases is still an open question for many companies. At the cutting edge of innovation, design agencies are becoming increasingly involved in co-design today. Direct face-to-face collaboration with end-users is a key factor for the success of co-design, and ICT solutions must therefore allow active involvement of all participants and their direct interaction. This paper reports an original evaluation of a Spatial Augmented Reality (SAR) environment dedicated to supporting co-design sessions. It investigates collaborative interactions and explores the role of design artefacts in the collaborative process. Observations of a conventional co-design situation are compared with those of sessions assisted by SAR technology. These two contrasting environments reveal both similarities and differences. The results show that within the SAR-supported environment there is an increase in end-users’ interaction density within the design activities. We observed that providing tangible artefacts augmented by digital content increased the density of interactions and encouraged greater sharing in co-design meetings.

1 Introduction

The design of industrial products takes place in extremely varied professional domains and requires the collaboration of multiple experts from different sectors ranging from highly technical to artistic. The study of Lahti et al. (2004), defines collaborative design as a dynamic form of communication that defines and consolidates design aims, explores the design space, discovers the associated challenges and constraints and, eventually, finds the appropriate solutions. Among the industrial practices of design, co-design, consists of the activity of co-creating the product with the participation of end-users as early as possible in the design process.

The involvement of external stakeholders (end-users, technical or scientific experts, subcontractors, etc.) has been an important factor in the success of concurrent engineering and design, and is still an important topic today (O’Neal 1993; Suurmond et al. 2020). Among the different types of external stakeholders, end-users have been considered by creative industries or architecture as a very relevant source of information, inspiration, and, eventually participant in the design of products (Caixeta et al. 2019). To facilitate the involvement of end-users, design approaches such as user-centred design and co-design have thus taken on new forms, where new actors are invited to become effective design activity partners (Sanders and Stappers 2014). However, the involvement of end-users represents a real challenge. Difficulties can arise from both communication and cultural barriers (Fussell and Benimoff 1995). To overcome such difficulties, and acknowledging that in design it is essential to be able to represent ideas, digital tool developers are constantly investigating new collaborative technologies (Ogara and Koh 2014).

Spatial augmented reality (SAR) is a type of augmented reality (AR) based on the projection of digital content on the surface of a white physical object. It has been considered as a promising technology to support face-to-face collaborative work and reduce the effect of the technology on direct interactions between the participants (Sereno et al. 2022). SAR can be used to integrate virtual objects into a physical environment, making their handling much more natural. For example, one can visualize a new layout design on a shoe and handle the object. Based on this technology, users can remain in their usual direct face-to-face environment while benefiting from digital interaction possibilities. Being able to use a physical space for communication purposes allows for conversation and natural connectivity.

As co-design calls for the participation of end-users in the early stages of the design process, designers have had to undergo a shift in mindset (where previously design was only a matter for engineers and specialists), methods and professional practices, subsequently leading to the rethinking of representational forms and human interactions. For example, end-users can be naturally integrated into design meetings thanks to new types of external representations offered by the SAR technology. This technology represents a useful opportunity to support the end-user participation.

In this paper, the impact of integrating a spatial augmented reality platform into co-design meetings is examined. The aim is to assess the extent to which the introduction of a new form of representation—where tangible aspects are combined with digital projection—influences co-design interactions and consequently the overall collaboration between the different stakeholders. A series of observations involving professional industrial designers in their working environment is presented. Their design interactions in a conventional setting are then compared with how they interact in a spatial augmented reality environment.

2 Co-design and information technologies in design: a state of the art

2.1 Definition of co-design

The term “co-design” is polysemic. According to Ulrich (2003), the term’s connotations differ with the context. Architects, software and systems designers use it to refer to the combined community, cooperative and collaborative design occurring among partners during a design project (King 2007). Anderson-Connell et al (2002), argue that “co-design” describes a collaborative relationship between consumers and manufacturers wherein, through a process of interactions between a design manager and a consumer, a product is designed according to the consumer’s specifications and based on manufacturing components. Another variant of these definitions is that “co-design” is a process that entails the participation of designers (i.e. experts who will make the product) and end-users (i.e. people who will potentially buy or use the product). “Co-design” can also generally designate the collective creativity taking place across the whole span of the design process (Sanders and Stappers 2008). In this sense, it is a specific instance of co-creation. Steen (2013) argues that co-design refers to the collective creativity of designers who combine creativity and inquiry Consequently, as it supports most of the definitions mentioned before, we follow Sanders and Stappers (2008) definition “We use co-design in a broader sense to refer to the creativity of designers and people not trained in design working together in the design development process”. Therefore, co-design is used to refer to the activity of designers and what we call “end-users” who work together as part of the design development process (Sanders and Stappers, 2008).

2.2 Involving end-users: from participatory approaches to co-design

Over the past decades, designers have been moving increasingly closer to the potential users of their designs. In areas where technologies are mature and additional value is difficult to obtain, industrial companies have become progressively open to approaches that accurately capture the users’ needs. User-centred design is a research field that includes approaches that require the active involvement of the potential users of a future product in an iterative design process (Maguire 2001). The user-centred design approach (i.e. ‘user as subject’), has been primarily driven by the introduction of social science in the design process. Since the 1980s, a gradual process has evolved where end-users have been allowed more influence and room for initiative. Gradually, end-users have been asked to provide expertise and participate more in early design phases by informing, ideating and contributing to conceptualising activities (Sanders 2002). By the turn of the 1990s, participatory approaches had become widely disseminated and were chosen as a subject of research in various fields such as human–computer interaction (HCI) or computer-supported collaborative work (CSCW) (Kensing and Blomberg, 1998). More recently Drain et al. (2018), proposed the application of this approach to developing countries providing another example of its usefulness for designers in non-conventional usage contexts with strong cultural differences between the participants (i.e. from north and south). Design thinking can also be considered as a new form of participatory design given that this approach is fundamentally user-centric and entails designing both the object and its use (Bjögvinsson et al. 2012). Participatory approaches have also been very useful and complementary to inclusive approaches where “lead users” play an important role. Besides, participatory approaches prevent, in the way the product is designed, certain categories of users (e.g. elderly or disabled) from being stigmatised (Wilkinson and De Angeli, 2014). The study of Luck (2003) reports on participatory design in architecture and also highlights the criticalities of involving building users in the design process by architects. Participatory approaches have also proven to be very promising in the development of new technologies, for example in the design of VR applications where the interface and interactions with the user are of prime importance for innovation acceptability (Bruno and Muzzupappa, 2010). Finally co-design designates the ultimate evolution of user-centred and participatory approaches by integrating the end-users in the creative and decision-making loop (Sanders and Stappers 2008).

2.3 Studies on mediated interactions

Today, it is broadly acknowledged that design has a social, technical and organisational dimension. The technical dimension requires knowledge, models, methods and tools. The social and organisational dimensions call for methods, rules, norms, standards, as well as internal and external regulations. But above all, design teams need tools to communicate and share information. Design artefacts have been the subject of various studies and many researchers assume that analysing these artefacts during design meetings can provide a more in-depth understanding of the design activity.

Adopting an ethnographic perspective on engineering and design practices, many studies have analysed how the objects being built relate to the designers at work (Jeantet 1998; Boujut and Blanco 2003). Vinck and Jeantet (1995) proposed the concept of Design Intermediary Objects to characterise the mediating role played by these artefacts in the design process. In line with Star’s Boundary objects concept (Star 1989), this concept of intermediary objects focuses more on technical practices and covers all the documents, drawings, virtual or physical models, sketches and digital 2D and/or 3D models produced and shared during design work. According to these authors, design intermediary objects have three main dimensions: they are representations (of both the cognitive and the physical dimensions of the design object); they are translations (from one or multiple intentions into an external object); and, finally, they act as important mediators between physical actors, conveying the intentions, decisions and knowledge of those having produced them. The role of these artefacts is thus very special while their form and content deeply influence both the design space and the collective work (Vinck 2011). Understanding the role of the intermediary object is important for this research as co-design situations such as those shown in Fig. 2 would normally involve intermediary objects. Other researchers have explored how the artefacts produced during the design process externalise the thinking of the designers and play an important communicational role (Petre 2004). Others focus on how a common ground can be established between stakeholders as a solution to improve mutual understanding and facilitate communication (De Vries and Masclet 2013). This grounding actually enhances co-creation between design actors through the shared representation of a current design problem or design discussion. Visser (2006), also investigates the role played by the common ground created during co-design activity. She explains that design actors can efficiently create shared representations once this common ground has been established through “concern agreements, especially on the definition of tasks, states of the design, references of central notions, and weights of criteria and constraints”. Many other studies have considered the concept of shared representations and their facilitating role in the design task. For example, Conklin (2006), argues that shared displays can help clarify disagreements in a workgroup. In other words, when ideas and concerns are mediated via a shared intermediary or boundary objects, this helps participants to clarify the reasons for their disagreement. Adopting a cognitive perspective, other researchers (Bettaieb et al. 2007; Lund et al. 2013), underline the importance of shared representations for understanding decisions and putting forward arguments in the assessment of design solutions. In their study, Boris and Whyte (2009), explain that in collaborative design these shared representations support knowledge and mediate collaborative knowledge creation.

2.4 Digital tools for collaboration: new technologies as solutions to foster collaboration

Considering that shared representations play a crucial role in creating a common ground for knowledge sharing in design teams and are apparently good candidates for mediating interactions between designers, it is reasonable to assume that studying artefact centric interactions will provide some interesting insight into the inner dynamics of design teams. As reflected in our research questions presented below in Sect. 3, we aim to analyse the various types of interactions occurring during co-design sessions. We shall especially focus on shared objects, these being physical mock-ups, drawings, printed prototypes, participants’ sketches, etc. Participants interact through these objects as they point at them, pick them up and modify them. Owing to the complexity of the collaborative design process, many solutions have been designed to assist stakeholders throughout the project. Applied to design, computer-supported collaborative work (CSCW) studies the requisite conditions for people to work collaboratively through technologies.

Collaborative technologies support various aspects of computer-mediated communication (general communication, information sharing, context management, group awareness, etc.) (Lonchamp 2003). Studies of Sadeghi et al. (2010) or Hisarciklilar et al. (2009) examine the use of specialised tools to assist collaborative design (e.g. a shared multi-touch table and a 3D annotation tool). These authors focus on communication aspects and how the use of these specific tools can facilitate information sharing.

More recently, a new family of software programs has been added to the designers’ toolbox. Virtual Reality (VR) tools and Spatial Augmented Reality (SAR) respond to the growing need for designers to overcome the difficulties related to viewing complex design models and making them accessible to a wide audience of potentially non-design experts (Cascini et al. 2020). However, natural face-to-face communication still remains challenging in these environments. Fussell and Benimoff (1995) and Dix et al. (1998) argue that natural communication is the ideal toward which computer-mediated communication should be directed. If we consider that natural communication entails all possible communication channels (sight, hearing, touch, smell and taste) (Ferrise et al. 2017), the ultimate goal of AR and VR ICT tools—as they strive to support the collaborative design activity—should be to ensure that communication is as natural as possible.

2.4.1 Virtual reality

Virtual reality is being tested in numerous industrial applications today. Conventional VR setups consist of a large screen on which an omnidirectional projector displays stereoscopic information. The system can support a high number of users, each equipped with passive glasses. The main disadvantage of this kind of system is that collaboration is limited to verbal exchanges between users. Participants do not share the same physical environment and their reactions to discussions remain hidden. Since the room where the system being used is dimly lit (to preserve the limited capacity of the projection systems), communication occurs under difficult conditions with all non-verbal arguments being restricted (gaze, facial expression, etc.), and personal items (notebook, notes, etc.) difficult to access. In addition, all interactions are usually directed by a single mediator or managed by a collective interface. Haijun et al. (2018) present Spacetime, a collaborative scene editing tool that enables parallel manipulation of objects to resolve interaction conflicts and support collaboration in an immersive virtual environment. Surale et al. (2019), for example, proposed a 3D-tracked multi-touch tablet in an immersive VR environment to accomplish 3D modelling operations. Their design space also includes mid-air gestures to interact with 3D content; however, it presented only a small subset of possible interactions. While such tools have improved the collaborative aspect, many of the VR solutions nevertheless remain limited and the head-mounted displays (HMD) are still very uncomfortable after a while. In conclusion, the VR approach generally favours the task rather than the participant. It can be considered as an inadequate collaborative solution since the user is equipped with peripheral devices preventing natural communication.

2.4.2 Augmented reality and spatial augmented reality

Azuma et al. (2001), define augmented reality (AR) as a technology where reality is combined with virtuality. AR is characterised by the introduction of digital content to a physical environment hence providing users with additional information. The study by Wang et al. (2016) highlights that AR solutions ensure context-awareness, in comparison to full VR applications, which represents an important asset of AR compared to VR. Nevertheless, it also has its limits, especially with respect to the number of users, each of whom must have their own HMD. Additionally, wearing an HMD makes it difficult for users to interact with the rest of the environment, especially with the other participants. Direct face-to-face communication is therefore limited. Billinghurst et al. (2008), investigate the difference between groups of designers working with traditional tools and others working with AR. The study assumes that using AR tools makes the task take longer compared with a face-to-face setting. However, the results show that more questions were asked when the designers used AR tools. The researchers conclude that AR technology is limited by the HMD and recommend the use of technologies offering more communication facilities such as hand-held AR devices or spatial augmented reality. In addition, the study by Franz et al. (2019) presents a double AR sharing technique, to enable couples of museum visitors to explore mixed-reality exhibits together. They assume that their AR tool improved communication and the sense of presence. However, there are limitations due to participants’ complaints about the discomfort of the use of AR head-worn displays (HWDs). Moreover, the classification by Bimber and Raskar (2005) presents that SAR technology does not physically interfere with users inside the environment, unlike head-mounted and hand-held AR systems. To overcome these limitations, SAR technology is based on the projection of an image on to a physical object which allows direct face-to-face interactions around a tangible artefact. It is also known as projective SAR (Furht 2011), or projection mapping (Fig. 1). The survey study by Sereno et al. (2022) confirms that SAR environments can be non-intrusive and then adopted for specific contexts. However, SAR still suffers some limitations such as colour rendering, the accuracy of displayed elements and—despite interesting efforts (e.g. Park et al. 2015)—this remains an open issue for industrial applications and widespread adoption.

Fig. 1

Spatial augmented reality application developed within the SPARK project

As a summary of the abovementioned works, the main argument for applying SAR to design is perhaps allowing design participants to communicate naturally. O’Hare et al. (2018) evaluate the impact of SAR on the novelty and quality of ideas produced during co-design sessions. Their study reveals that SAR enhances both, but entails a highly complex setup process. These results confirm the conclusions from Akaoka et al. (2010). They highlight that SAR environments require an important amount of implementation and configuration work in order to prepare the design sessions, but that participants are satisfied with their interactions in such an environment. The participants confirm that the interactive hands-on approach and the ability to quickly change components in a SAR environment are important features to support co-design.

In 2013, Irlitti and Itzstein (2013) underline that SAR technology offers flexibility since the physical object in the scene augmented with digital content enables immediate modification of the design representation and real-time feedback by all participants. Their study is based on gathering feedback from three design experts on the added value of applying SAR technology to a co-design activity. The study by Ben Rajeb et al. (2014) focuses on the impact of SAR on collective reflection in design projects. It highlights how “SAR participates perfectly in group cohesion by creating intermediary spatialities between augmented presence and virtual co-presence”.

3 Research gap

Despite the encouraging results of the above-mentioned research, it should be noted that a limited number of tools have been tested with users in real conditions. There are few reports of experimental results able to support these preliminary findings. Additionally, very little research in the field of collaborative design has involved multi-disciplinary groups with external stakeholders (O’Hare et al. 2018). Our experimental approach is part of the SPARK project, introduced in the following section, and represents a first step towards assessing the level of participation in face-to-face collaborative design meetings using SAR technology.

4 Research question: does SAR technology improve co-design interactions?

Through the act of communicating, people can build and transform their knowledge (Thomas and McDonagh 2013). Communication in co-design sessions is characterised by comparing and contrasting different points of view through verbal argumentation and gestures, but not necessarily involving design artefacts as a form of mediation. In their study, Coyne et al. (2002) confirm that a shared drawing device, visual contact (gaze direction), information sharing and the use of a drawing board affect how designers work. Therefore, in design, communication channels can include speech, eye contact, gestures and design artefacts. These artefacts and design representations play a key role in the way the design task is approached. Several studies agree that the design of complex products occurs mainly during the negotiation and argumentation phases between participants, through proposals, essays and evaluations (Bùdker 2000; Eckert et al. 2000). Studying the interactions between designers can therefore offer more insight into design collaboration. These studies point to the need to focus on artefacts in the design process in order to develop environments allowing direct interaction between participants and the design representations. SAR technology can create a digital-physical design artefact that would support the development of such environments. Our aim here is to concentrate design on how collaborative interactions can be improved in a spatial augmented reality environment. Based on the above-mentioned research, we have chosen to focus on the impact of the technology on interactions between different participants during co-design sessions. We shall investigate the extent to which the spatial augmented reality tool affects end-users’ participation. We will use the density of interactions, that is represented by the cumulative number of interactions per participant as a metrics to evaluate the participation in the design sessions. Our research seeks an answer to the following two sub-questions:

RQ 1: Does SAR technology substantially improve the density of interactions in co-design sessions?

RQ 2: Does SAR technology substantially improve the involvement of end-users in co-design sessions?

5 The spatial augmented reality environment for this study

The aim of the European project SPARK (SPatial Augmented Reality as a Key for co-creativity),¹ was to develop a new tool for design representations, that would make it possible to reduce the number of prototypes made while allowing physical interactions with objects during co-design meetings. The resulting tool is a SAR platform comprising different sub-systems to be used in design sessions. The platform operates in real-time by combining projected digital design elements on a physical prototype. It can be used to facilitate the brainstorming phase of a design project as it enables faster generation and review of design solutions, avoids misinterpretation, and at the same time accelerates the production of design representations (O’Hare et al. 2020).

The platform enables design actors to interact around a shared object based on a rough shape of the intended product (Fig. 2). This shared object is called a mixed prototype and is defined as the combination of a physical prototype and a digital projection (Morosi et al. 2018). In a previous paper, the authors demonstrated the use of the platform in the domain of the packaging industry and proposed interesting performance metrics for the co-creative sessions (Cascini et al. 2020). The platform acted as the testbed for the study reported in this article. Therefore, the research question we address in this paper is a continuation of the SPARK project and aims at getting further in the understanding of the role of the artefacts in design collaboration.

Fig. 2

Co-design meeting supported by the SPARK platform

6 Definition of the analysis framework

Although several studies have already explored interactions in design meetings, they essentially aimed at studying the cognitive impact of gestures (Visser and Maher 2011; Mcneill 1994). Our analysis framework has been designed to assess the respective involvement of design participants during co-design meetings and the impact of the technology used on this involvement. Therefore, in what follows, we shall define a taxonomy of gestures associated with artefacts. We consider that any action accomplished by a participant and directed towards another participant, either by means of speech, gesturing or an object present in the design environment, constitutes an interaction (Ben Guefrache et al. 2018). This interaction can involve just one or two of the means mentioned above or involve a combination of all three. We aim to assess the proportion of artefact-mediated interactions compared with pure speech-based interactions. In light of the study by Tory et al. (2008), we define interaction modalities as cognitive or physical engagements with artefacts. As explained by Tang and Leifer (1988), while design artefacts are important for collaboration so too are the mechanics of the interactions with these artefacts. Some modalities require the physical engagement of the participant, such as viewing, gesturing or pointing at and handling. Our framework considers three elements (Fig. 3). Combining all these aspects and based on the elaborated literature (presented in Sect. 2.3), we define three main dimensions of design intermediary objects:

• Who initiates the interaction? Actor category

• What type of artefact does the interaction involve? Artefact typology

• Which modality does the actor call on to interact with the artefact? Interaction modalities

Fig. 3

Gesture analysis framework

6.1 Typology of artefacts

We have four categories: tangible, digital, mixed and ephemeral. They correspond to the type of technology and design artefacts involved in our research.

• (a) Tangible artefacts

  The tangible artefact category covers all physical design representations present in the meeting room and used during the discussion. It includes physical mock-ups, prototypes, drawings, sketches, printed representations and any notes written in a notebook or on a post-it or paper. In collaborative interactions, a tangible artefact may be used by participants (end-users or designers) in any manner. Interactions supported by tangible artefacts are coded when participants talk while holding them or simply look at them while talking.

• (b) Digital artefacts

  The digital artefact category includes any kind of media displayed on a surface such as a TV screen, laptop, tablet, smartphone or any ICT tool. This kind of artefact can be a presentation including text, pictures or images of the product, which is the most frequent case, or videos. A digital artefact is involved in collaborative interactions when participants (end-users or designers) point to the surface, handle the device while searching for information or look at the information displayed on a screen while talking.

• (c) Mixed artefacts

  As defined in Sect. 4, mixed artefacts can be defined as physical objects (e.g. 3D printed physical mock-ups), on which digital content is projected. This content can be pictures, images, text or textures. The mixed artefact is referred to as such because of its composite nature, i.e. physical and digital content are combined in the same representation. This kind of artefact is typical of SAR technology. A mixed artefact is involved in collaborative interactions when participants (end-users or designers) point to its surface, hold or handle it, or simply talk about it while looking at the information displayed on it. According to Vinck’s classification (2009), mixed artefacts are considered as open intermediary objects since the digital content can be modified, discussed and improved in real-time.

• (d) Ephemeral interactions

  During co-design, interactions not involving any type of artefact (digital or tangible) can be observed. In this case, participants communicate through what they say and how they gesticulate. This category encompasses all types of air gestures identified during observations. Two main types have been identified for the purpose of this research: virtual artefacts and communication gestures.

  • Communication gestures correspond to hand gestures that may replace or accompany speech. They have no task-related content. They can be considered as a beat accompanying the rhythmic pulsations of speech (Eris et al., 2014). Their role is to underline or modulate the speech itself. This category has been added to facilitate coding by differentiating them from the content-related gestures involved in the design task described below.

  • Ephemeral artefacts are imaginary objects depicted or mimicked through air gesturing. For instance, the gesturer may depict or mimic the shape, volume or surface of an object, or simulate its use (a function in a specific context), or behaviour (e.g. the flashing of a light, the pressing of a button, etc.). The gesturing may simulate an action performed by an object or an action carried out by a person on the object. Virtual artefacts are suggested via iconic gestures (e.g. an idea can be illustrated through gestures imitating the movement or shape of real objects tied to speech) or metaphoric gestures (e.g. an idea can be explained through gestures imitating the movement or shape of concepts) (Cash and Maier, 2016).

• (e) Without artefact

  When an actor interacts without referring to an artefact, this is coded as “without artefact”. Concretely, no artefact is used when participants simply talk to each other and look at each other without any other form of support (they do not gaze at an artefact, designate an artefact or hold an artefact in their hands).

6.2 Typology of interaction modalities

Interaction modalities depict the kind of action performed on the artefact. They may depend on the design environment; especially the technology used, or vary according to the actor. This is why they have been included as a second level of coding in our analysis framework.

• (a) Pointing

  Pointing occurs when a hand, finger, pen or any tool is used to point directly at an artefact or to a specific part of a prototype or content displayed on a screen (deictic). Pointing has many different meanings in the field of social communication (Tory et al. 2008). However, in an environment using ICT tools, most of the time pointing is used to select a particular item on an interactive surface (Kendon 1996). However, in our research, no interactive surfaces were present owing to the type of artefacts used. The pointing interactions observed were therefore deictic.

• (b) Gesturing

  Gestures are defined as interactions involving physical engagement. Gesturing implies the use of hands and other parts of the body for communicative purposes. Many studies also define gestures as a complement to speech with their aim being to express what cannot be expressed through words. For example, a product’s functions or behaviour can be simulated through gestures or speech supported by certain movements. Our analysis framework takes into account air gestures to facilitate communication and gestures related to the design object (see Sect. 5.2.1). For example, hands can be used to simulate the turning of a knob or the pressing of a button. The objects may be displayed digitally and the gestures relate to the design task to be accomplished.

• (c) Viewing

  The viewing modality corresponds to the action of gazing at an object or digital content. It can also be used to point to an object and indicate a location.

• (d) Manipulating

  The manipulation modality corresponds to all types of interactions with a physical object that involves picking up or grasping the object. This modality applies to actions such as touching, lifting, holding, carrying, dropping, etc. It also includes modification, annotation and sketching actions.

Considering the presented framework, our coding strategy used for gathering data consisted of multiple steps. First, we identify the collaborative interaction (or main interaction). We define an interaction as collaborative when it is at the core of the co-design activity and is focused on product development. This interaction involves at least two participants; if two interactions co-occur the collaborative interaction is the one that involves the highest number of people. For example, two participants look at the mixed artefact and at the same time another participant writes down on his notepad, the collaborative interaction is the one with the mixed artefact; the secondary interaction with the notepad is not taken into account. The second step is to define which type of artefact is involved in the collaborative interaction. The last step is to consider which modality is used to interact with the defined artefact. For example, an artefact is identified as at the core of the collaborative interaction when a participant points at a given artefact. ‘Pointing’ is then identified as the interaction modality.

6.3 Discussion on shared interaction modalities

Based on the presented definitions of design artefacts (Sect. 2.3), interaction modalities contain aspects that help design participants to collaborate. Shared interaction modalities is a category that is composed of pointing, gesturing, manipulation and viewing interactions. A personal interaction occurs when a participant interacts with an artefact without any communication form or shared interaction with the other participants.

The importance of the shared interaction modalities is confirmed by Mondada (2006), who asserts that pointing in a co-working setting puts a common spotlight on the objects pointed at. Pointing captures the attention of other participants and focuses interactions on the object being pointed at. Tory et al. (2008) claim that when sketching annotations are shared, they enable the group to mentally visualise the proposed structure. The works by Pfeuffer et al. (2016) and Qvarfordt and Zhai (2005) explore how the users’ gaze can support collaborative interaction whilst using digital tools. These studies highlight the role of eye gaze in collaboration. Additionally, many studies consider gestures as a communication tool (Visser, 2010; Visser and Maher, 2011; Cash and Maier, 2016). Others such as Lebaron (2000) and Yasui (2013) examine the role of gesture repetition in a joint activity context. They demonstrate that gesture repetition helps to build a shared knowledge among participants. These studies confirm that gestures play an essential role during collaborative idea construction. In reference to the above works, “shared modalities” will be considered here as communication and knowledge sharing facilitators. Shared modalities will be analysed against personal interactions to evaluate whether SAR facilitates sharing compared with a conventional co-design environment. This proposed second level of our coding scheme will enable us to investigate the level of collaboration by counting the occurrences of shared modalities. By studying collaboration in this way, we also get closer to our research question concerning the contribution of end-users in co-design tasks.

7 Observation protocol

7.1 Description of the case studies

The Design Agency (TDA) is a consulting company specialised in product design. This company is the unique source of case studies presented in this article. The products TDA designs are manufactured products for different industries ranging from small start-ups to international groups. They have an office in Spain and an international one in China. For the purpose of our research, four case studies were selected from the TDA portfolio on the basis of the following criteria: the object of the design task were products or parts of them with comparable level of complexity and manufactured B2C products.

The four case studies were co-design sessions slightly different in terms of task and objectives. However, the aims of each session were to review and evaluate some design proposals prepared by the designers which implement ideas from a previous meeting. All the observed cases were in the ‘ID definition phase’ (Fig. 4) where the design agency provides a solution that responds to the requirements. In this design phase, physical prototypes allow the user to manipulate and interact with the product. In addition, digital contents are often displayed to show other design options.

Fig. 4

Description of the ID definition phase

7.1.1 Description of the products

Barbecue case: the design task was mainly to review the latest developments of the barbecue design. This review was based on the available prototype (and others from competitors) and on some design files provided by the design agency. The meeting rapidly focussed on the ash tray and the prototype was used to discuss several manipulation issues with the customer. In this session, the necessity to manipulate the prototypes was a clear justification of the selection of this case in our study. This is the first conventional case as presented Fig. 5.

Fig. 5

Characteristics of the four case studies

Beacon case: the design task here was to discuss colour and finishes in relation to the function of the object. Indeed, as the product is an emergency device to be used by mountaineers in critical situations, the colour of the artefact plays an important role as well as the material to securely handle the product. Special attention was given to the texture of the rear rubber case, which has been inspired by the contour lines you can find on a topographical map which indicate slopes and gives an extra visual and tactile aspect. The design agency discussed the choice of a double injected plastic base made out of a combination of an ABS plastic and thermoplastic rubber to achieve an IP67 watertight case which can endure the toughest situations. The participants could manipulate various sizes and shapes on the physical prototypes. Digital contents were also available. This is the second conventional case.

Ultrasound case: in this session the design agency proposed different alternative solutions based on the use of the SAR technology to display various scenarios of colour codes that the object could take depending on the information captured by the sensor. The product is a sensor using ultrasonic technology to detect infant bacterial meningitis. This is why it is important for the device to display colour codes when some critical zones are detected. For this co-design session, the digital contents were displayed simultaneously on a screen, and on the mixed object through the SAR technology. The participants used a doll to represent the patient and perform more realistic manipulation of the sensor.

Impedance case: in this session again the SAR technology has been used. The product was a prototype of a case for a Line Impedance Stabilization Network (LISN). LISN is fully compliant to CISPR 16-1-2 that facilitates the simultaneous measurement in both lines and the extraction of common- and differential-mode conducted emissions. Starting from two design proposals, the session aimed to establish a final layout of the elements on the front-end of the case (switches, connectors, etc.). This has been a co-creative session where different alternatives have been dynamically discussed and a final version has been elaborated on the basis of these discussions. The session was run fully on the SAR environment and the digital contents were dynamically moved around on the mixed object.

We have selected four products that were in the same level of maturity in order to ensure better comparability of the cases. In the ID definition phase, physical prototypes are presented to the client. They are not fully functional but display some shapes or partial functions. Lots of parameters need to be defined and there is room for decision and improvement. In this phase, the end-users can give valuable input on the basis of the partial prototypes. The ID definition phase seemed the most appropriate to fully take advantage of the SAR technology. Besides, in the traditional sessions (i.e., without SAR), the prototypes were at a comparable level of maturity as those used in the SAR cases. Additionally, the complexity of the design tasks and the technical contents discussed in the four sessions were comparable. The participants discussed colour, finishes, interface display and shapes. All these elements have a visual and ergonomic impact on the product while being rather technical and therefore requiring some technical expertise. Although the technological complexity of the devices is significantly different the discussions were intentionally focused on similar elements related to the use and perception of the product which make the cases comparable.

7.1.2 Description of the participants

Barbecue case: the actors attending the meeting from the design agency were the creative director and the chief engineer. Both of them are designers. The client was the CEO Europe of the barbecue’s brand who is also sales expert.

Beacon case: the actors attending the meeting from the design agency were the creative director, the business developer and the chief engineer. All three of them have a strong background in design. In this session, they worked with the CEO of the satellite-tracking device brand who is a marketing expert.

Ultrasound case: the participants attending the meeting from the design agency were the creative director and the design engineer. The client was the founder CEO of the medical device brand who holds a PhD in Biomedical Engineering. He has expertise in Medical Imaging and holds a master’s in Business Administration.

Impedance case: the session involved the CEO of the start-up who is also the creator of the technology and two experienced designers from the design agency.

In the four sessions, the end-users were represented by people from the client companies that have a direct connexion with the usage of the product. All had a background in technology and a marketing or sales additional competency. Even though they had different positions in the company (some were start-uppers others were employees) all shared a responsibility to represent the voice of the customer. On the design agency side, only three persons have been involved, two designers, the business developer and the creative director. They all have a very similar profile and background and have been working together on a daily basis for many years. All this is summarized in Fig. 5.

7.1.3 Display of the SAR and conventional rooms

The display of the conventional sessions’ rooms was slightly different as the size of the barbecue made it impossible to have it on the table (Barbecue case). Consequently, the participants had to stand up and move towards the prototypes to manipulate it, which was not the case for the other conventional case study (Beacon case) (Fig. 6).

Fig. 6

Display of the conventional sessions’ rooms (Barbecue case left, Beacon case right)

The display of the SAR room was the same for Ultrasound and Impedance cases. Only the mixed prototype varied (Fig. 7). The SAR functionalities were the following: the display of digital content on the mixed artefact (a white 3D shape of the artefact) using two video projectors situated on the top of the room over the participants. The mixed artefact was tracked so that the 3D contents was following the movement of the artefact accordingly. The users could visualize and manipulate the object as if it was a real product.

Fig. 7

Display of the SAR room at the company’s premises

In the four cases, the physical display elements were relatively close together: three to four people were sitting around a table with some shared physical artefacts and a computer screen to display digital contents. The only major difference was for the Barbecue case where people had to stand up and move around the table to reach the prototype.

7.2 Definition of the observation variables

Our research questions aim to assess the influence of integrating SAR on co-design interaction density (RQ1) and, more specifically, how this integration impacts end-users involvement (RQ2). The ultimate goal is to find out whether this technology encourages end-users participation and/or interaction during the co-design meetings.

The main variable used here is the technology. This is an independent variable and has two values: SAR and conventional. The impact of this independent variable on other dependent variables describing interaction density will be appraised. These variables were described in Sect. 5 on the theoretical framework and are summarised in Table 1.

Table 1 Definition of variables

The controlled variables which have been discussed in Sect. 6 have been carefully checked to minimize their impact on the overall results. As we were observing real industrial cases it was impossible to reproduce the same experimental conditions. However, as discussed before, the number of participants, the nature of the design task, the design phase and the physical setting have been carefully controlled.

7.3 Validation of observation protocol: pre-tests sessions and intercoder reliability

To ensure validity and reliability of our research protocol (Bryman 2001), we have carried out pre-tests in our lab environment based on industrial case studies. We also performed intercoder reliability tests to ensure reliability of our coding of the real sessions.

• (a) Pre-tests

  To validate the observation setup and protocol, we systematically ran 3 pre-tests for each condition. To ensure all the required aspects were covered in the capture phase (capture and post-processing) two main questions were asked. Firstly, what did we need to capture through our observations? Secondly, what would be the best means of capturing the data sought?

  Our answer to the first question was that we needed to gather all audio and video information and record all design activities carried out during the session. The best way of doing this seemed to be to record entire sessions and hence capture all video and audio information that would allow us to perform an exhaustive analysis. This method presents a major advantage in that, following the observation phase, multiple reviews and analyses of all the data gathered can be performed from different research perspectives. Additionally, the video recording allows the researchers to easily identify which participant is speaking and to whom and whether an intermediary object is being used or not. The video method also facilitates the transcription of verbal interactions. Furthermore, filming multiple views of the design activity makes it possible to reconstruct a complete view of the design session after it has taken place. The pre-test sessions took place in different environments: first in our lab (Fig. 8a, b) and later on in the Design Agency’s premises (Fig. 8c). The aim was to test the equipment to be used during the real sessions. In addition, the observation variables defined could also be validated during these pre-test sessions. Hence, we were able to ensure that our capture process enabled the subsequent analysis of all the interactions occurring, and that all the actors and artefacts involved were properly identified.

• (b) Intercoder reliability

  According to Blessing and Chakrabarti (2009), double coding is important to validate the results gathered. As these authors put it, “Double coding involves coding of at least a part of the data by two different people or by the same person twice but with a time delay in between”. It can thus be assumed that having a second coder affords a more objective look at the coding taxonomy and hence leads to a more reliable coding process and results. Intercoder reliability is an indicator of measurement consistency. Although intercoder reliability does not ensure validity, it is a crucial component in content analysis. Cohen's kappa (Cohen 1960) is a common method for estimating the reliability of nominal data. It is a statistical tool for measuring inter-rater agreement of qualitative (categorical) items. For our purposes, two test groups comprising two coders (three coders in total) performed several iterations with training in between in order to obtain an acceptable Cohen’s kappa. The following Table 2 presents our results:

Fig. 8

Pre-tests in Grenoble INP lab (b) in PoliMi lab (c) SAR platform configuration and test in TDA premises

Table 2 Results of our intercoder-reliability

The intercoder reliability results for the two test groups were better for the second video extract. This supports the importance of double coding and allows us to validate our coding procedure. Once the reliability test was considered acceptable, the coding procedure was set and the coding of the four videos was completed by one coder.

8 Results and discussion: the influence of SAR on gestural interactions in co-design sessions

Our theoretical framework focuses on the role of artefacts in co-design sessions. Through the observations, we aim to highlight the differences in the two conditions we have studied (i.e. conventional and SAR). Our observation framework captures the interactions occurring during these sessions. Conventional and SAR conditions are systematically compared using the dependent variables defined in Table 1, namely:

• Number of interactions

• Type of interactions

• Interaction modalities

• Sharedness of the interaction modalities

• Type of participants

8.1 The use of artefacts in co-design sessions

Our analysis shows that in both situations, artefacts occupy an important place whatever the type of object being handled. Artefacts are involved in almost 85% of the interactions recorded in both types of sessions.

As Table 3 shows, the interactions performed without artefacts are limited to a small percentage (up to 15%). In the conventional situations, only ~ 10% of interactions did not involve artefacts. In the SAR sessions, only a maximum of 15% of interactions was performed without artefacts. Given the high percentage of interactions involving artefacts, it can be said that design actors rely heavily on design artefacts during all design activities. The smaller number of interactions in the Barbecue case compared to the Beacon case is probably due to the fact that the participants needed to stand up and move around the table to reach the physical prototype. The SAR platform did not lead to a significant increase or reduction in artefacts used. Since artefacts are already widely used in conventional environments, we can conclude that the introduction of an ICT tool does not significantly influence their presence and involvement in design interactions. This supports the results reported in our state of the art (Sect. 2.3) regarding the importance of boundary objects and intermediary objects in all co-design sessions (Boujut and Blanco 2003; Star 1989).

Table 3 Involvement of artefacts in co-design sessions (all participants)

A deep look at the distribution of the end-users’ interactions (Table 4), shows a similar sessions preview. The involvement of artefacts is still important, however, the use of the SAR prototype does not increase it significantly in all sessions.

Table 4 Distribution of End-users interactions

8.2 End-users’ vs designers’ respective participation/engagement

Here we present the average of participation for each design actor (Table 5). The case studies presented involved the same number of designers and end-users.

Table 5 Number of different participants’ interactions in the co-design sessions

The table illustrates that in both cases, designers dominate the interactions: they initiate at least 57% of interactions in the conventional setting and 50% in the SAR setting. Whatever the environment of the meeting, designers are the most active participants. This is not surprising since they are the ones who lead and manage the discussion of the product. However, end-users’ involvement is significant as it accounts at least for 27% of total interactions in the conventional setting and 34% in the SAR setting, i.e. typically a small increase of 7% if we cumulate the two case studies in each condition. This finding suggests that SAR technology does tend to engage the end-users more in the design activity (up to 50% in the second SAR session) which supports our second research question (RQ2).

8.3 Ephemeral interactions

In Sect. 5.1.d, we introduced the notion of ephemeral interactions as a category of our analysis framework. Artefacts can be either normal communication gestures supporting speech or ephemeral artefacts simulating a component or function of the product. Whatever the case, these artefacts provide a visible trace of intense communication during sessions. The study by Davis confirms the key role of gestures in design activities. He underlines that to communicate their ideas, designers accompany their sketches with gestures, which are used as communication tools (Davis 2016).

In relation to all other interactions, interaction with ephemeral artefacts account for 14–21% in the conventional setting and 27–35% in the SAR setting (Table 6). We can suggest that conventional sessions show an acceptable number of gestures in the air as a means of communication. However, the number is higher in sessions involving the SAR tool. The SAR platform would therefore seem to offer an environment that encourages participants to be more expressive and communicative > 27%) since they gesticulate more in this setting. It can thus be argued that collaboration through gesturing is stimulated thanks to the presence of this tool.

Table 6 Ephemeral interactions vs all other interactions (tangible, digital, mixed) in the co-design sessions by all participants

A closer look at the results in Table 7 shows the general trend highlighted in Table 6. For the Ultrasound case, we observed that the end-users naturally integrated the external object provided (baby doll) to simulate product use. The baby doll was used to position the device and simulate the different scenarios. During the session observed, 23% of all the interactions by end-users involved ephemeral artefacts. In the Impedance case, the front panel of an electronic device was displayed and different switch positions were tested by the users who then gave their feedback by using gestures in the air to simulate the use of the switches (40%). In both situations, the end-users’ involvement was significant; they were seen to interact spontaneously with the mixed artefacts, performing gestures to simulate the use of the product. This kind of interaction was less pronounced in the conventional setting where the shared artefacts generated less feedback perhaps due to the static nature of the physical prototype (a barbecue for the first case study and a static casing for the Impedance case). In line with what has been highlighted before we can argue that SAR stimulates the interactions between the participants which supports our first research question (RQ1).

Table 7 Ephemeral interactions vs all other interactions (tangible, digital, mixed) by End-users

8.4 Distribution of shared and personal interaction modalities

Sharedness of the modalities, as defined in Sect. 5.3, cover the pointing, gesturing, manipulation and viewing that take place during an interaction involving the participants. A personal interaction occurs when a participant interacts with an artefact without any communication form or shared interaction with the other participants.

Table 8 shows the distribution of shared vs personal interactions in both settings. As can be seen, both display a very high proportion of shared interactions, which is understandable given what we observed were co-design meetings. However, the level of personal modalities is particularly low, which indicates that the participants spend very little time thinking on their own while interacting with an object. This result is also supported by Stempfle and Badke-Schaub (2002), who observed little self-reflection (i.e. thinking about the task itself) in their study of collaborative design. Our case study observations of the SAR setting revealed a slightly higher proportion of shared interactions = (98% and 96%) in comparison to the conventional environment (87% and 96%). This difference is not really significant especially if we consider the interrater reliability index of maximum 0.72. We would need more research to investigate whether the SAR environment fosters more collaborative interactions. This result should, however, be compared with the type of task observed. The aim of the sessions observed was to gather feedback from end-users about design alternatives. In other words, little idea generation was expected. In the meetings we observed—and in line with our theoretical framework—the mixed artefacts used in the SAR sessions seemed more successful as intermediary objects than other types of artefacts. They may thereby better serve the desired effects of this type of session. This is an interesting line of investigation for future work.

Table 8 Comparison of the number of shared and personal interactions by all participants

8.5 Density of interactions

This section investigates if the design participants perform more interactions while using the SAR tool. Table 9 presents: the total number of interactions (personal and shared) in both conditions; the sessions’ duration (sessions were ended when the designers considered that all tasks on the agenda had been completed); and the density of interaction in each session. The density is the ratio of the number of interactions divided by the duration of the sessions. The density of interactions allows us to normalize the results and make them more comparable than absolute numbers that of course depend on the duration of the session.

Table 9 All participants’ interaction density in the two conditions (personal and shared interactions)

As can be seen, the duration of sessions supported by the SAR platform was half that of the conventional sessions and yet generated more interactions. This result clearly shows the substantial effect of the SAR setting on interaction density. Since the aim of the sessions was to get good user feed-back, we can consider that more interactions lead to more information provided to the designers. In that sense the SAR setting appears to create the conditions for more productive and dynamic sessions. The designers benefit from the information and feedback needed from users in a shorter time.

Table 10 depicts the end-users’ contribution in both settings. The same trend as that shown in Table 9 can be seen for end-users’ interactions, i.e. an interaction density between 2.4 and 3.5 in the SAR setting as opposed to 0.91–0.97 in the conventional setting. Sessions supported by the SAR tool demonstrate a higher end-users’ density of interaction which tends to support our first research question (RQ1). In comparison to the previous indicators, there is no significant variation between end-users and designers, so it does not disproportionately favour one stakeholder over another in a co-design session.

Table 10 End-users' interaction density in the two conditions

8.6 Contribution of the study

Immersive approaches in virtual reality environments do not allow direct face-to-face interactions. This interaction is only possible via digital avatars in virtual worlds. This is clearly a limiting factor that spatial augmented reality environments do overcome by allowing direct interaction between the stakeholders. On the other hand, the expressive power of virtual reality environments is often put forward with its ability to create narratives and story-telling (Rubio-Tamayo et al. 2017). Additionally, the capacity to represent non-figurative elements such as data, models or concepts is clearly an important positive factor of virtual reality. It seems that the major separation line is the direct face-to-face interactivity. When the benefit of having a direct face-to-face interaction, or a more realistic interaction environment with tangible objects is greater than the benefit of having a sophisticated representational environment, it is certainly better to choose a spatial augmented reality solution over a VR one. In return, if the representation and the immersive context allows the simulation of a rich and sophisticated environment that is not possible to attain in a real setting, then virtual reality offers huge potential.

Concerning the use of artefacts, these results show that participants’ behaviour in the SAR and conventional settings is very similar. In both, design artefacts are involved in most of their interactions. Our qualitative observations show the benefits of the mixed artefact maybe that: participants can pick up and move a mixed object as naturally as they do in a conventional situation where they interact with conventional tangible objects. Combining tangible objects with the projection of digital elements seems to encourage the participants to keep handling the prototype while also allowing them to modify it in real-time and put forward their suggestions. The main difference observed between the SAR setting and the conventional one comes from the slight improvement in end-users’ involvement. As already suggested, being able to change the intermediary object (mixed artefact) seems to encourage end-users to be more expressive compared with static objects in a conventional environment. One limitation is that we have no means to determine whether this enhanced involvement is due to the nature of the mixed artefact (combining tangible and digital elements) or the excitement associated with the first-time use of the technology (a possible ‘wow’ effect). However, our qualitative observations from the videos do not show any significant differences in the behaviour of the participants. The sessions were carried out in the same professional manner, with a high concentration on the task. The novelty of the environment did not appear to influence the general behaviour of the participants.

As suggested above, the same trend can be observed regarding the participants’ communication behaviour. Indeed, ephemeral artefacts are more prevalent in the SAR environment. Based on our review of the literature, we have used these gesture-related artefacts as a positive communication indicator. In other words, in this sense, SAR technology has a positive effect on communication between design participants. We have also explored the effect of SAR on interaction modalities. In general, the SAR setting produces results that are slightly different from those of the conventional setting. However, the conventional setting already favours the use of shared modalities (Table 8). Our SAR findings present little difference with respect to the number of shared modalities and the reduction in the use of personal modalities. In terms of interaction density, our observations reveal a significant difference in the rate of interactions occurring in the SAR sessions (Tables 9 and 10). It can thus be argued that using SAR technology has a very positive effect on information flow and feedback between participants. Overall, these findings show that the SAR platform has a positive impact on the participants. It allows the designers to pick up more information from the end-users who are able to interact naturally with the SAR content. Collaboration is facilitated by the sharing of a real-time editable design representation consisting of a physical prototype with the projection of digital elements.

9 Conclusion

In this paper, we have explored co-design through the lens of design interactions in conventional and spatial augmented reality (SAR) environments. In these two observation settings, conventional and SAR-equipped, we assessed artefact-centric interactions as a means of assessing collaboration between design stakeholders. Physical interactions were considered as the most objective observable elements of the design meeting as they do not require invasive observation methods hence keeping the situation as natural as possible. This method also did away with the need for interpretation, which is the case with speech analysis for example. The observations in this research are based on real industrial situations reflecting the everyday design work in a design agency. We have chosen to observe four design sessions of significant duration with professional designers in their working environment.

The first results for main research question (Does SAR technology improve co-design interactions?) are encouraging. This study allows us to claim that providing tangible artefacts augmented with digital content seems to enhance the participation of end-users and designers while not perturbating the design process itself. Combining artefact-based interactions and a mixed prototype seems to be a promising interaction interface allowing manipulation and real-time modifications simultaneously. The findings suggest that: SAR technology does tend to engage the end-users more in the design activity (up to 50% in the second SAR session) supporting RQ2; and the measure of interaction density is substantially higher in the SAR sessions thereby supporting RQ1.

Besides, the profile of the sessions in terms of interactions do not vary significantly, suggesting that the design activities are not drastically different in the two conditions. Therefore, we can claim that whatever the environment, the technology does not impair the concentration of the participants who remain focused on the task rather than disturbed by the novelty of the technology.

The study also shows that the SAR tool is capable of integrating end-users in the design process with ease and effectiveness, especially because participants simply have to sit around a table as in a conventional meeting and can interact face-to-face in a natural way. We observed that the end-users were participating more during the SAR sessions which tends to demonstrate that more interactivity associated with a physical artefact is beneficial to the integration of external stakeholders. We confirm that tangible design artefacts play an important role in that they support interactions in both situations and confirm what Brereton and McGarry (2000) found in their study of engineering students engaged in design sessions. Tangible objects reveal properties and limits of the hardware that are not present in digital environments. Additionally, their findings show that physical objects are important media for communication, as they are used to command attention, persuade or demonstrate.

Through this research, we also verify existing understanding of the importance of artefacts in the design process. Indeed, in our study, only a very small proportion of design interactions occur without the support of an artefact (physical or virtual). The interaction density is one clear indicator of the advantage of the SAR environment, the trends observed are encouraging and more research needs to be performed in this field.

The advantage of focusing on gestures is that they are an external manifestation of cognition and hence a good indicator of the performance of the different external representations provided by new technologies. However, an approach paired with speech analysis might offer some interesting insights into the cognitive dimension of collaboration. As suggested in their study with autistic children, the use of an interactive object enabled the implementation of a “co-design beyond words” approach where the authors were able to pay attention to micro-interactions that can unveil design insights conveyed beyond words (Wilson et al. 2019). This extreme approach clearly shows the potential of studying gestures associated with design interactions and encourages further research on co-design with technologies such as spatial augmented reality.

We could study how the mixed representations contribute to identifying more issues, requirements, ideas, etc. It would be interesting to study to what extent the design activity itself is transformed by the technology and the relationship between gestures and cognition in these digitally augmented environments.