Keywords

7.1 Introduction

The concerns of Architecture, Ubiquitous Computing and Human Computer Interaction research have begun to overlap. Historically, this has been enabled by computing moving from the desktop into the environment via the emergence of ubiquitous and pervasive computing. Technically, this enabled sensors, actuators, processing and the interfaces to these to be embedded into the fabric of our surroundings, originally designed to function invisibly and to free us from performing mundane tasks (Weiser 1991). Because of these developments, researchers now address Human Computer Interaction in a much broader, considering the environment and artefacts as in tangible computing (Ishii and Ullmer 1997). Rogers frames this development as moving from users to context, employing multi-method study approaches, integrating knowledge from multiple disciplines to develop engaging user experiences that are evaluated through a value-focused lens (Rogers 2009). As a consequence, HCI research, drawing on Ubiquitous Computing technologies, is now frequently occupied with understanding interaction in the environment, a concern that Architecture traditionally holds.

As Coyne has recently re-stated, Architecture completely un-augmented by computation is already highly interactive (Coyne 2016). This is not to say that architects have not so far considered the inclusion of sensors, actuators and processing in their buildings. Quite the opposite is the case, and office buildings for example have been equipped for a long time so that their indoor climate can be tightly controlled, this history having been traced by Banham (1984). Today, eco-homes are commonly fitted with computer-controlled equipment with the aim to support people in reducing their carbon footprint and in creating a healthy and comfortable living environment. In addition, the rapid growth of the Internet of Things, results in ordinary homes to be augmented with digital technologies on an even larger scale. Beyond these more common examples, architectural research today includes interaction enabled through computation as an elementary part of its design palette. This has lead to a set of related publications addressing interactive (Fox and Kemp 2009), responsive (Bullivant 2005) and robotic (Bier 2014) architecture, which can be summarised as constituting the field of Adaptive Architecture (Schnädelbach 2010). Some of the key properties of such architecture are briefly described in what follows.

Sensing embedded in the environment but also body-worn (and communicating with the infrastructure embedded in the environment), provides information about people’s location, movement, physiological, mental and psychological states, and their identity. Actuation in the architectural environment can be concerned with the light and sound infrastructure, environmental controls, data flow and media displays, resource supply and architectural components and elements, including their movement. In principle, anything that can be sensed about people can be linked to actuations in the environment. When such actuations are made, a feedback loop emerges between people’s behaviour and the behaviour of the environment. Such feedback loops have been demonstrated in eco home research (Hong et al. 2016) as well as bespoke lab experimentation (Schnädelbach et al. 2012).

One focus of Adaptive Architecture research is the interaction between human movement and movement present in the environment. It is timely that this is considered in more detail, as architecture includes kinetic elements in more cases and sensor systems to capture people’s movement are becoming more capable and widespread. As the number of prototypes that link human movement and architectural movement increase, the likelihood of such designs emerging in everyday buildings increases. In this context, it is essential that Architecture and Interaction research develop a better understanding of the opportunities and constraints that this brings and this chapter contributes to the development of this knowledge.

7.2 Background

The following briefly reviews existing work in mapping human and architectural movement.

7.2.1 Movement in Architecture

Everybody knows—and especially architects, of course—that a building is not a static object but a moving project, and that even once it is has been built, it ages, it is transformed by its users, modified by all of what happens inside and outside, and that it will pass or be renovated, adulterated and transformed beyond recognition. (Latour and Yaneva 2008).

While Latour’s essay frames this as a representational problem, i.e. he is mainly concerned with how we might capture the fact that buildings are not static, works by Duffy (1990) and Brand (1994), have explored the more practical sides of which aspects of the built environment change over time, with those of larger scale and shared moving less rapidly than those smaller items that belong to individuals. In recognition of the non-static nature of buildings, Habraken has encapsulated this in his ‘Supports’ strategy, combining mass-production of supporting frames with individually adaptable dwelling units to enable adaptation over time (Habraken 1972).

There is also a class of buildings that are specifically designed to be mobile and the history of portable architecture has been succinctly captured by Kronenburg (2002). More recently, drawing on technical advances in production and control, the emerging kinetics of buildings and building components has been considered by Schumacher et al. (2010), who demonstrates how wide-spread such approaches have now become in the built environment. For a more generalised overview of this space, the previously mentioned framework categorises possible movement in buildings as changes of location, orientation, to building form and topology, changes to building components and to the relationship of inside and outside (Schnädelbach 2010, p. 7). Examples of movement occurring in buildings include those that offer moveable internal partitions, moveable separations of indoors and outdoors, moveable building units such as rooms on wheels, but also various types of moveable roof structures, among other possibilities.

7.2.2 Movement in People

Without exception, all expressions of human behaviour result from motor acts (Solodkin et al. 2007), and movement is therefore our main way of engaging with others and the environment. This emphasis on movements is also present in those approaches to cognition, which emphasise the embodied and embedded nature of our presence in the world (Varela et al. 1991; Wheeler 2005). Here, our bodies and the environment are seen as continuum and we cannot but leave traces in it, a fact that Richard Long has specifically explored in his environmental art (Long 1967).

Previous work that has considered human movement in the context of Adaptive Architecture focussed on the scale, expressiveness and control of movement (Schnädelbach 2016), which we briefly summarise here. The scale of movements, also used by Abawajy in their taxonomy (Abawajy 2009), ranges from micro to macro movements. This is in turn related to how visible their effect might be to an external observer: the internal movements of the cardiac muscle are relatively small scale and in many occasions invisible to others. In contrast, the muscles in our legs allow us to produce our largest movements through space and this movement clearly becomes observable. This visibility and related to this the expressiveness of specific movements are directly linked to legibility by others (‘reading’ someone’s behaviour and their psychological state via their body movements). In the context of this chapter, human movement must be expressive to and legible by whatever sensing system is employed, as this detectability is required to make links between body movements and architectural movements. Finally, there are different levels of control that people have over their body movement. Some movements and movement patterns (e.g. breathing) are controlled unconsciously and consciously. Most of the time, people don’t think about their breathing, while they clearly can for example for relaxations purposes (Montgomery 1994). Other movements are much more clearly aimed and targeted, for example when controlling fine-grained grasping actions (Solodkin et al. 2007).

7.2.3 Movement-Based Interaction

Human movement, and considerations of its scale, expressiveness and control, has been part of HCI research from the outset, while not necessarily its focus. Standard interfaces like the computer mouse and the touch screen, but also less standard technologies like head mounted displays, take human movement as input for the interaction with computing. The PUC special issue on movement-based interaction is testament to the persistent interest in this area (Larssen et al. 2007), proposing that the moving body should be considered as part of any interaction and considering the new design spaces that emerge. Over the last decade or so there have then been a number of endeavours to frame movement-based interaction design more specifically. Loke et al. have provided an overview of such frameworks in Loke and Robertson (2013) and Cruz provided an updated, more comprehensive listing recently (Ricardo Cruz et al. 2015). In what follows, we draw on these overviews to synthesize the central elements for consideration in movement-based interaction design but only as they are relevant for this chapter.

Human movement requires (a) space. The considered space is shaped by the human movement of interest to the designer and it might for example be sized to accommodate a single user’s hand or accommodate the full body movement of multiple people. This space also becomes shaped by interactive technology, as both (Eriksson et al. 2006) and (Schnädelbach 2012) have outlined in the case of interactions supported by camera tracking and video communication, respectively. The particular (b) interactive technology used in movement based interaction design also influences the design more broadly. Body-worn motion sensing offers very different affordances to sensors embedded into hand-held devices or camera-based motion tracking and the relationship of sensing and interaction has for example been considered by Benford et al. (2005). At the heart of the concern is the actual (c) interaction to be designed in the sensed space, linking human movement to interactivity of some kind. A chosen set of body movements is sensed and used to drive a system. This might link movements of our hand to movements on screen, it might link two tangible devices together as in the inTouch prototype (Brave et al. 1998), it might amplify physiological data (Marshall et al. 2011), or it might involve whole body movements. Finally, (d) movement in people that can be sensed in the given interaction space and is relevant to the desired interaction is what drives that interaction. Loke et al. describe how movement has become a new design material in this context that needs to be more fully understood by changing one’s own practice (Loke and Robertson 2013). In parallel, it is equally important to understand what movement can be detected best by what technology and what the key properties of available technologies are.

7.2.4 Movement-Based Interaction in Architecture

There is a wealth of movement-based interaction design in architectural history, going back nearly a hundred years. Rietveld Schröder’s house (Kronenburg 2007, p. 26) offers physically adaptive features such as moveable partitions that allow the manual reconfiguration of the interior, an idea still relevant as demonstrated in Holl’s Fukuoka Housing (Kronenburg 2007, p. 52). Naked House by Ban (Kronenburg 2007, p. 170) takes this a step further by providing room units equipped with wheels that can be freely placed within a larger domestic volume. Beyond entirely manual engagement and the interior, much larger scale movements have been implemented for example in Studio Gang’s Starlight Theatre (Studio Gang Architects 2009) and DRMM’s Sliding House (DRMM 2009). In these cases architectural movement is driven by motors, which are triggered by consciously controlled human interactions to trigger an architectural change when desired.

In experimental architecture emerging over the last decade or so, enabled by the parallel development of applicable sensing and actuation technologies, relationships between movement in people and movement in architecture have become more subtle. TU Delft’s Muscle Tower (Hubers 2004) is programmed to physically react to the proximity of people that share the same interaction space on an immediate level, Alloplastic architecture makes use of a Kinect tracker to link full body movement to the movement of a tensegrity structure (Farahi Bouzanjani et al. 2013), while Slow Furl is developed to map human presence on a much slower scale (Thomsen 2008). The more interactive couplings on this spectrum have recently been captured by work on architectural robotics (Bier 2014).

The ExoBuilding (Schnädelbach et al. 2010) and Breathe (Jacobs and Findley 2001) explorations present another take on movement-based interaction in architecture, as they make use of physiological data to actuate a physical enclosure of an interaction space. The examples across history demonstrate how human movement at various scales, expressiveness and levels of conscious control has been amplified or attenuated to be mapped into architectural movement. Despite this wealth of previous work spanning Ubiquitous Computing, Human Computer Interaction and Adaptive Architecture, there is a lack of knowledge of what such environments mean for their inhabitants. While there is continuing and growing interest to propose and develop movement-based interaction to deploy it in everyday settings, a subset of Adaptive Architecture, the growing community of collaborating architects and interactions designers lacks a full description of the design and interaction processes and the potential consequences of the designed interactions.

7.3 MOVE Platform

MOVE has been developed with the specific aim to explore the relationship of body movement and movement in Adaptive Architecture. As a platform it aims to allow for the following: (1) Physical configurability: to enable different architectural configurations (2) Mapping configurability: to enable flexible mappings between human movement and prototype movements (3) Non-expert use: by creating an interface that allows 1 and 2 to be done by non-programmers. The overall aim was to create a re-useable research tool for different contexts. In what follows, we briefly describe the MOVE platform and its development process before focusing on the interaction with one particular instantiation of the platform, prototype 3.

MOVE consists of a software platform and physical components and actuators, which can be arranged in space. Two floor-to-ceiling poles are used to mount four structural arms each, on which rotatable panels are mounted. The arms can be adjusted in height and they can be fixed in any rotated position around the cylindrical poles. Height constraints are therefore given only by a combination of panel length (considering that they might rotate toward the floor and ceiling) and floor-to-ceiling height. Panels are made from foam core board and have large cut-outs to save weight and to reduce air drag. The shape and size of the actuated components can easily be adapted. They are mounted to short rotating arms. Each panel assembly is directly fixed to the output axis of a model servomotor, which in turn is affixed to one of the mounting arms. The panels are weight-balanced to an extent to take the strain off the motor mounts. The active movement range of each panel is approximately 180°.

Using two Phidgets (Phidgets INC.) 8-channel servo controllers, four moving panels were implemented each side (with expansion to sixteen panels physically and programmatically possible). As can be seen in Fig. 7.1, panels were arranged symmetrically on the prototype, with four each side, mirroring the symmetrical nature of the human body, but not its anatomy. Asymmetrical arrangements would be possible.

Fig. 7.1
figure 1

The 15 body joints (left) as seen by OpenNI (Graphic derived from Nanoxyde GFDL (http://www.gnu.org/copyleft/fdl.html) and MOVE prototype (right) and panel layout (Top level panels L1 and R1, medium level panels L2 and R2, lowest pole mounted panels L3 and R3 and floor standing panels L4 and R4)

Full size image

A Processing Foundation (2016) project used the Phidgets21 library (Greenberg and Fitchett 2001) to interface with the Phidgets hardware and the SimpleOpenNI library to interface with the Microsoft Kinect V1. The Kinect is located at the forward centre of the MOVE prototype facing inwards to a point roughly between the two poles, but one metre out (compare Fig. 7.1). A user interface created through Swing in Netbeans allows interaction designers to make appropriate links between body movements and prototype movements, but also has the functionality to record and replay MOVE component movements. A full description of this interface is included in section ‘Prototype version 3’. The functionality of the hardware and software platform allows a single person’s body movement to be tracked where each of the eight panels is associated with a specific limb movement (see Table 7.1).

Table 7.1 Mapping of the degrees of freedom (DoF) of the upper and lower limbs of a person (feet and hands are assumed rigid here) to the degrees of freedom available in MOVE (drawing on Herman 2007). The 26 degrees of freedom available on the left and right hand side of the body are mapped to the eight degrees of freedom available in the MOVE prototype L1-4 and R1-4
Full size table

7.3.1 Prototype Version 1

Drawing on Borenstein et al. (2012), the first version of the implementation tracked the distance between body joints, as OpenNI sees them. As illustrated in Fig. 7.1 there are 15 points that are technically trackable and useful for interface programming, while they are not all anatomically correct.

Translating body movement to prototype movements involved calculations of the distance between two joints, the resulting dynamic value of which was individually mapped to pairs of panels. For example, the distance between the right hand and the right shoulder of a person was mapped to panels R3/L3 in version 1. On either side of the body, this version of the prototype additionally tracked the distance between elbow and torso joint, the distance between knee joints and torso joint and the distance between hand joints and the centre of mass of the respective body side. Tracking four sets of relative joint distances allowed the mapping to the four sets of two panels, so that the entire prototype could be actuated. Table 7.1 describes the particular, chosen mappings in the context of all Kinect trackable movements and available MOVE movements. A person’s shoulder pitch is mapped to panels L1 and R1, shoulder yaw is mapped to panels L2 and R2, Elbow pitch is mapped to L3 and R3, while Knee pitch is mapped to L4 and R4.

We evaluated this first functional version of MOVE in a demonstration and focus group session (lasting around 90 min) with five colleagues from our lab, to feed information forward into the development process. A short presentation was followed by a demonstration and a session in which each of the participants tried the prototype. Unstructured discussion during this trial was followed by a recorded semi-structured focus group. A first enthusiastic response from participants and statements about empowerment (the prototype made one participant feel bigger) was tempered by concerns about ethical data use and privacy. In particular, the ‘nervous’ nature of this first implementation (the servo motors always being ready to engage and slightly shaking) would have made people uncomfortable during longer spells of use. The mapping between body and prototype was also not seen as accurate and predictable enough. Participants requested a better representation of the body movements mapped to the prototype as they found them hard to comprehend. The discussion included suggestions for additional mappings, for example the reversal of directions of movements from body to prototype, and a better interface to the possible mappings, including all configuration options. With regards to applications, participants discussed how the prototype would be useful to training in martial arts and dance and may be theatre where the audience could have input too.

7.3.2 Prototype Version 2

Further development focussed on addressing the tracking issue. Rotational sensing was implemented where the angles between limbs were tracked, instead of the distance between joints. This provided for improved tracking accuracy and stability in practice. Four participants (P1-P4) were recruited at University to experience an introduction to MOVE, a short demonstration and an individual trial session with the prototype. This was followed by a brief semi-structured interview. There were males and female, all between 20 and 25 years old. We briefly summarise the resulting formative feedback.

Movement—Understanding: We observed how participants moved to acquaint themselves with the prototype, movement itself being the learning strategy. They tried things out, seeing MOVE react, adjusting to what it can do, and for some participants this translated into a form of exercise. Participants requested more trackable movement types, for example the idea to track and adapt according to the torso direction was mentioned.

Connectedness, Speed—Delay: Perceived delays and comparatively slow speeds of the prototype meant that only certain movements gave participants a sense of connectedness. We observed how this could lead to adaptations in behaviour, so that participants moved in ways that they had learnt could be tracked by the prototype.

Inaccuracies—Jitter: A certain jitteriness of the prototype is clearly noted, caused by the rapid cycle of reaction and counter reaction in the digital servos used, which keeps those in position and ready for engagement. Inaccuracies in tracking are also evident to participants, as the sensor does not handle self-occlusion or rotations away from frontal view very well.

7.3.3 Prototype Version 3

For the final prototype version that was then studied, we replaced all remaining digital servomotors with analogue alternatives. This simple change in hardware reduced the jitter in the prototype considerably, and the slight loss in reactiveness of the engines was not substantial.

Beyond this smaller change, refinements included experimentation with more trackable movement types, such as the distance between the two arms, the distance between the two legs, hip pitch and torso rotation. While these were not used in the trial of the MOVE prototype described below, they prompted the creation of a more complete design interface to the movement mapping available in the prototype, as shown in Fig. 7.2. The interface allows the eight trackable body movements to be mapped to the 8 (possible) pairs of servomotors, giving access to control over their velocity, acceleration as well as their range.

Fig. 7.2
figure 2

MOVE interface to mapping possibilities. The interface lets designers map body movements to prototype movements, allowing adjustments to panel velocity, acceleration, as well as min/max positions. Configuration can also be saved here

Full size image

7.4 MOVE in a Martial Arts Context

For a trial of the final version of the prototype, we invited two Tetsudo practitioners in two sessions. Tetsudo is a martial arts, which emerged in the 1960’s, drawing on other martial arts forms (Dhaliwal 2016). It focuses on the unity of body and mind and emphasises self-control over self-constraint. This results in the aim to avoid being angry and aggressive when there is no need and avoiding to be passive and hesitant, when there is a need to act.

In practice, Tetsudo is contact-less and has a substantial performance aspect to it, enacted through Tetsudo ‘Kheds’. These are ‘…set pieces of ‘imaginative conflict theatre, in which the artist immerses himself/herself to experience the full array of the physical, emotional and intellectual dimensions of a ‘conflict situation.’ Dhaliwal (2016). Teaching is conceptual, rather than prescriptive, so that no two Tetsudo practitioners perform Kheds in exactly the same way. Both individual performance and sparring with a partner are important parts of the practice. We invited Tetsudo practitioners because of their focus on own body movement, movement skills and expected ability to reflect on body movement when providing us with feedback.

7.4.1 Method

The Tetsudo practitioners (P5 + P6) participated in two trial sessions. P5 attended the first session (Tetsudo 1) alone. He has practiced Tetsudo for around 8 years and wears a purple belt. P5 joined P6 in the second session (Tetsudo 2). Her experience is of more than 15 years and she wears a double black belt. The two experimenters were present throughout but mostly out of sight of P5 and P6. Sessions were recorded with 3 video cameras, one positioned at the back, one at the side and one at the front. A semi-structured interview was recorded with P5 following his session and with P5 + P6 following their joint session. Video and audio were transcribed across the sessions and interviews. We draw on these transcriptions for the descriptions of the session structure and for the development of analysis themes, which are used below to describe the interaction with MOVE in the context of Tetsudo.

7.4.2 Overall Session Structure

The experimenter introduces the mapping of body movements to each pair of panels, moving from L1R1 to L4R4. Following this, the participants are asked to complete four MOVE postures: Close all panels, open all panels, close right hand side and close left hand side (see Fig. 7.3). Both participants complete this warm up period in approximately five minutes without major difficulties or concerns and it provides them with a base understanding of the prototype capabilities and how to manipulate MOVE with their bodies.

Fig. 7.3
figure 3

MOVE—Open and closed with the corresponding body postures by the participant

Full size image

These short introductions are followed by individual ‘free’ sessions, in which both participants explored Tetsudo movements and postures in relation to the prototype. Participants 5 and 6 spent around 20 min and 10 min respectively in those individual free sessions. The character of the two individual sessions was different, in that for P5 it was their main activity during their trial session. For P6, it was the first half of the session, leading into a shared part (P5 and P6 working together; see section Shared session—P5 and P6 below). However, P5 and P6 both had sufficient time with the prototype by themselves to get fully accustomed to it. Deliberately, no detailed instructions of what to do or achieve were provided for the free session by the experimenters. As for P1–P4, the aim was to let them explore the possible interactions with MOVE, but this time specifically framed by their Tetsudo experience and their learnt movement repertoire.

7.4.3 Individual Session 1–P5

The individual free session of P5 begins with a bow, the same way that a Tetsudo session starts (Fig. 7.4). P5 continues the session with a wide-opening ‘ready’ posture, as illustrated below. Standing still, both arms move up and side-ways to come back down again in front of P5. This movement is particularly well tracked. P5 returns to a short pause and this set of movements in this session, whenever tracking has been poor, or when to begin trying something new (Fig. 7.5).

Fig. 7.4
figure 4

Bow following warm-up and at start of free session

Full size image

The first part of the session is characterised by a quite methodological exploration of what postures P5 can achieve working with MOVE, referring both to postures that MOVE can exhibit and those that P5 can demonstrate. This includes keeping still, with the expectation that the prototype would remain still. When it does not, this is being noticed and being checked, as illustrated below, when L4 continues to move although P5’s left leg remains stationary.

P5 also trials upper body movements standing on one leg, crouching down, moving arms synchronously and asynchronously, crossing limbs over and keeping them separate and facing forward as well as away from the Kinect sensor, Tetsudo kicks and punches, executed passively (slow) and vigorously (fast). For the last three minutes of the session, P5 concludes with a section using Tetsudo sticks. At this point, interaction becomes most deliberate, possibly framed by the constraints the sticks put on the interaction. P5 crosses arms over less frequently.

7.4.4 Interview

During the interview P5 began to reflect on the movement relationship to MOVE, stating that there:

… were some moves, which would kind of bring it all together … . There was certain postures … it was very reflective of that … . We do something called a ready posture … you start almost everything with this. It was nice to feel that responsiveness. (compare with Fig. 7.5).

Fig. 7.5
figure 5

Sequence of Tetsudo ready posture with P5 raising arms and MOVE open (left) and P5 with completed ready postures and MOVE mostly closed (right)

Full size image

P5 continues to describe the lack of responsiveness in other circumstances when he reflects on the speed with which MOVE can track a performer:

It is a little linear … and I think the pace, the speed is …, if I did things rapidly, it couldn’t respond to it … . So I stuck to moves that it was responding to, ….

P5 reflect on the fact that this might suit a Tetsudo beginner quite well, as students start out repeating quite linear movements. However:

… when you get more … advanced, it’s much more kind of around … the circular movements and spinning kicks and things like that and I think it [MOVE] would … possibly not respond so well to that.

7.4.5 Individual Session 2–P6

The second individual Tetsuo session had quite a different character from the first. P5 invited P6 to join in the second trial, roughly five weeks after the first trial, proposing P6 because of her superior Tetsudo skills. In the intervening period, P5 would have had the opportunity to explain the prototype from his perspective and his interaction with MOVE. Such explanation is clearly present in the individual session of P6, when P5 first proposes to do Khed movements passively and vigorously during the session, but then adds: ‘… it [MOVE] struggles with … the speed … it manages to cope better, if you are moving slowly.’ While P5 stresses that there is no right and wrong and proposes P6 explores the whole range of Tetsudo, P5 clearly frames the MOVE range of capabilities for P6. P5 remains in the space with P6 during this second individual session, outside tracking range, and P5 and P6 occasionally discuss during the session and also involve the experimenters when they have specific queries (Fig. 7.6).

Fig. 7.6
figure 6

P6 showing fine-grained control over MOVE in a number of set poses following each

Full size image

P6 begins the session with a broad range of Tetsudo moves, before a short focus on leg kicks. Both passive and vigorous Khed movements are visible, with the faster movements showing considerable delay in the MOVE response. About seven minutes into the 10-minute session, a sequence of Khed movements lasting for around 50 seconds, best demonstrates the level of control that can be achieved working with MOVE. The fact that control has been achieved is confirmed by P6: ‘Feel like, I have got them (the panels) tamed now.’.

7.4.6 Shared Session–P5 and P6

Following the individual trial by P6 during the Tetsudo 2 session, both P5 and P6 used MOVE in a joint session, which lasted for 26 min in total. Both participants were in control for parts of the session, with the respective other participant standing on the opposite side behind the Kinect sensor, joining in or just observing (see Figs. 7.7 and 7.8). The session was interspersed by discussion, P5 and P6 discussing what they wanted to try out, occasionally confirming with the experimenter. Broadly, the session can be split into three parts: During a first part, Tetsudo freestyle sparring lasted for approximately ~7:30 min, with P6 being tracked. During the middle part lasting ~6:50 min, the session is characterised by trialling expressive postures. In a final part, lasting for ~9 min, P5 and P6 mirror each other with MOVE mirroring the lead participant. The three parts are further described below.

Fig. 7.7
figure 7

P6 Lurching forward towards P5 (left) and Move following, pointing at P5 (right)

Full size image
Fig. 7.8
figure 8

P6 creating intimidating MOVE posture (left) and P6 ‘hissing’ at P5 through MOVE (right)

Full size image
Fig. 7.9
figure 9

P5 mirroring P6—Stretched out (left) and crossed over (right)

Full size image

7.4.6.1 Freestyle

Tetsudo distinguishes between compromised, competitive and combat freestyle, successively increasing the level of contact with the opponent and the force of the contact (Dhaliwal 2012). All are with partners. Compromised is aimed at helping in training, making use of Kheds learnt and sequencing those. Competitive (light contact) and combat (firm contact) are scored in competitions. P5 describes freestyle as ‘action and reaction’, with one person reacting to opportunities to strike, which the opponent created by leaving open space for attack. Both action and reaction movements are aligned with a set repertoire of recognised movement sequences or Kheds. In the first part of the joint session, MOVE mirrors P6 with P5 engaging from around 3 m away, standing behind the Kinect sensor, as described earlier. Using MOVE, freestyle here seems to be most similar to compromised as a large gap is introduced, i.e. there is no contact between opponents.

During the first two minutes, freestyle sparring proceeds at regular ‘Tetsudo’ pace’, which is generally too fast to be tracked accurately by MOVE. Prototype movements appear quite erratic and sparring seems to ignore the prototype in some sense, with participants fully concentrating on the sparring partner. This prompts P6 to suggest: ‘Shall we go slower, do you think …’ with P5 agreeing to that suggestion. During the two minutes that follow, freestyle sparring continues at a considerably slower pace, with much better movement mapping by the prototype, and P6 confirms this by stating: ‘I am working on making my reaction to P5 much more clearer, rather than too complex.’.

From an observer’s viewpoint, sparring now appears much slower and the interaction with MOVE is becoming part of the Tetsudo routine in a way that seems to ‘make sense’ to both participants. However, MOVE adds something to the Tetsudo practice, which P6 describes during a brief in-session discussion as follows:

I am fighting P5, well or working with P5, with a distraction (pointing at MOVE) … not so much a distraction actually, with an extra little thing to think about. It’s like … you are doing a dance and saying your five times table or something … You know, doing two things, a physical thing and then an awareness thing. And … I quite liked it.

The above highlights two distinct influences of the MOVE prototype on the performers’ practices. First, they adapt the speed and ‘clarity’ of Kheds to allow MOVE to follow them, which has a slowing effect and it makes movements more legible. In addition, for an experienced performer, MOVE seems to add an extra layer of complexity to their movement practice. This complexity results in a worthwhile challenge for P6 and a possible extension to the Tetsudo practice per se, challenging a person through two concurrent embodied interaction patterns.

7.4.6.2 Expressive Postures

A brief discussion at the end of the sparring session results in P6 experimenting with making MOVE more clearly part of their ‘attack’, drawing the panels into be part of the action by P6, which P5 would have to respond to (during this short episode, P5 is only observing). Following this, P6 lurches forward in a throwing movement (see Fig. 7.7 left), which result in the MOVE panels lurching forwards towards P6 (see Fig. 7.7 right), albeit at reduced speed.

This results in P5 stating (pointing at panels L2 and R2): ‘Yeah, these two are intimidating.’ and P6 continuing with another noticeable body posture as illustrated in Figs. 7.8 and 7.9. This is commented on by the two participants:

P6: :

That’s everything up.

P5: :

I mean, it’s, in terms of uhm, that, what you had it just then, it’s, it’s quite a you know, powerful sort of gesture … that kind of like the way you had it then.

P6: :

Yeah

This posture is carefully repeated by P6 before the conversation continues. P6 structures her body posture for maximum effect in the MOVE posture, re-enforcing the creation of an intimidating expression, while her own posture is not necessarily intimidating. P5 reacts to MOVE’s expression with an expressive posture of his own, raising both arms and spreading out all fingers. He states: ‘Yeah kind of like … (making an aggressive hissing sound)’.

This demonstrates the potential for people to amplify their body postures through MOVE in an expressive way and the ways that this is perceived by a counterpart as expressive. It also shows a deliberate strategy of making the prototype ‘fight’ on one side, with one of the participants opposing the other.

7.4.6.3 Mirroring

At the beginning of part 3 of the shared session, P5 takes over to be tracked. He demonstrates his mastery over the prototype through controlled movements, resulting in very good tracking. Initially observing only, P6 then begins mirroring P5, before suggesting that they continue the session in this fashion. This is agreed and it leads to a three-way connected system: P5 exhibiting relatively slow, deliberate Tetsudo movements, with MOVE and P6 mirroring those movements (see Fig. 7.9). From the observation it seems clear that both parties know the movement sequences, which makes it possible for P6 to mirror P5 well. A roughly two-minute section of this is followed by a brief discussion of symmetry, with P5 expressing that he thought that symmetrical body movements are more successful than moving only one side of the body. Participants then swap over for P6 to be tracked, with mirroring continuing, exhibiting both one-sided and mirrored body movements, confirming this in a brief discussion:

P5: :

So, where you doing it the opposite, so if I was doing it with my left hand (raising left hand), you were doing it with your right?

P6: :

Yeah.

P5: :

I just mirrored everything that you did

This second mirroring sequence produces some of the calmest and best-tracked sequences with clear evidence that P6 fully understands the tracking range of MOVE.

This section concludes Part 3 of the joint Tetsudo 2 session. It demonstrates how MOVE can be used as a dual-mirror, it following one participant and the second participant following the first and MOVE itself.

7.4.7 Interview

The semi-structured interview following the shared session probed participants’ reactions to the prototype, to the appearance of the prototype, desired extensions, possible applications and possible opportunities and concerns. In what follows, we briefly reflect on the three core themes emerging from the interview: interaction, limitations and uses in the Tetsudo martial arts practice.

7.4.7.1 Interaction

The participants discussed how a moving environment like MOVE would draw people into explore more, out of curiosity. This is because encountering MOVE presents the challenge of working out what connections there are and how they work; what moves in relation to what and establishing a link to it as P6 explains:

I suppose it’s trying to make that link between something that’s totally separate to you.

Reflecting comments made during the trial session, P6 also re-iterates the fact that MOVE presents a challenge once a link has been established:

… it’s really fascinating to have this extra dimension that you’re controlling. It challenges a different part of your brain when you’re moving.

The fact that there is reaction of some kind is what supports the establishment of a link between participant and MOVE, while the particular appearance of the prototype is not important in this as expressed by P5:

I think the physical appearance of this is not important. More so the fact that it’s responding to the movement.

Beyond visual feedback, sound is mentioned as a contributing factor to creating a specific connection with the prototype, as explained by P6:

I think when you’ve got an audible link with the movement it instantly adds another connection with it. So, if you took that sound out and it was completely silent … there’ll be another level of connection, whether it will be a deeper level of connection or a lighter level … It’s the senses, isn’t it? It’s how you’re sensing that piece of equipment.

Through this comment, participant 6 highlights the importance of taking a holistic view in the design of feedback environments.

7.4.7.2 Limitations

The two participants agreed that they were looking for more familiarity of the prototype using it first, and that this might have delayed making a link with the prototype initially, and P6 states:

I think we’re always trying to make something familiar, so you’re trying to make that look like two shoulders, two elbows. And that’s why I feel that we struggled with the knee one [referring to panels L4 and R4].

Neither of the two participants directly suggest a more anthropomorphic appearance of the prototype, but something where mappings are more easily legible and where those mappings more directly reflect a person. The above also points to the fact that the movement range of the prototype was seen to be too limited. While L4 and R4 where mapped only to the movement in the knee, participants expected those panels to more closely follow the movement range of the whole leg, which would have enabled the mapping to kicks, for example. The two participants continue to discuss this more specifically in relation to Tetsudo. In particular, the lack in variation in speed was mentioned. As highlighted earlier, Tetsudo includes rapid and slow movements and MOVE cannot express both as P5 outlines:

… from a Tetsudo perspective, … we have the sort of … range of … styles from very passive movements to very vigorous movements, and at the moment I think it only really responds to the very passive, slow movements.

This results in vigorous Tetsudo movements (combining strength and speed) not to be represented at all, providing limited expressiveness. A simpler, but very important limitation was the participants’ expectation for the prototype to be still, when they perceived themselves to stand still.

7.4.7.3 Role in Martial Arts

The final theme concerned the risks and opportunities of the application of such a prototype in martial arts. With the focus on the visual and external, participants saw a risk in loosing the focus on one’s own body, which is essential to Tetsudo training, as P6 stated:

… where you’re putting the onus on something more visual to give you, to help you with your movements, your physical body movements, I think you’re taking away something. You’re taking away mobility that this one human being normally does …

This is discussed as part of the more general observation that people are increasingly dependent on technology in all aspects of life. At the same time, a prototype such as MOVE would have great potential during the training of Tetsudo as P5 concludes:

… as a part of training, I think something like this would be a really interesting thing for someone to do … when they’re learning Tetsudo … It would become an integral part of the training.

7.5 Summary of Findings and Discussion

Kinetic Adaptive Architecture is increasingly being built around us, as evidenced in the introduction and background section. Within this field, movement in architecture can be related to movement in people and a smaller set of prototypes are now proposed that have this interactional capability, providing new forms of interaction. This chapter has drawn on its broad, multidisciplinary context and the iterative development and evaluation of the MOVE prototype. This lets us now reflect on how the feedback loop between person and environment shapes and limits interaction, and how the selectiveness of the MOVE ‘mirror’ becomes useful in practice and training. We conclude with an outline of the co-creation process in this context: (1) Designers of movement-based interaction embedded in Adaptive Architecture need to draw on and design around the correspondences between person and environment. (2) Inhabiting the created feedback loops result in an on-going form creation process that is egocentric as well as performative and embodied as well as without contact.

7.5.1 Feedback Loop Between MOVE and Body Movements

The movement range of MOVE is clearly limited in comparison to human movement and this is also evidenced throughout the trial feedback. The Kinect sensor does not track some presented movements. For example, the rotational movement in the lower arm is not being seen by the Kinect, and hands and feet are seen as solid. As demonstrated in the trial sessions, Tetsudo includes untrackable movements such as finger movements, but it also includes untrackable relationships of body movements, such as the crossing of limbs. Tracking technology continues to evolve, and the use of Kinect version 2 would have resolved some of the above issues. However, there will be tracking technology limitations for some time to come, which means that human movement tracking will remain incomplete in practice.

There are only 8 (4 pairs) of single degree of freedom movements that MOVE can produce, even though they appear to be much more complex when executed in unison. Given that feet and hands are seen as solid by the Kinect sensor used, participants are presenting 26 trackable degrees of freedom to the camera (13 on each side (compare Table 7.1)), when facing forwards. Unlike human movements, which are fundamentally integrated with each other (Bernstein 1967, p. 22), there is also no interrelation between the movements that MOVE can produce, all panels being actuated and mapped independently from each other. Whereas the two Tetsudo performers deliberately varied the speed of their interaction, as it is part of their practice, the MOVE prototype only coped with the slower and less forceful movements. When fast movements were followed by a short pause, MOVE was able to catch up. Through a sequence of fast movements, MOVE was not able to keep up. The experienced slowness results from the overall system performance chain, including tracking performance, motor acceleration and speed, as well as panel inertia.

The differences in appearance, the tracking issues, lower number of degrees of freedom and slower speed of MOVE in comparison to a person, result in participants and prototype having very different embodiments and capabilities. An anthropomorphic robot was never under consideration for this work, as the emphasis was on the investigation of the relationship of human and architectural movement. For this reason, overcoming the correspondence problem (Nehaniv and Dautenhahn 2002) was not a specific aim (and it seems clear that solving the correspondence problem is currently not possible, even if that was desired).

Importantly, the lack of correspondence in embodiment between participants and MOVE does not lead to a breakdown of the experience or the abandonment of Tetsudo in this particular context. Instead, the practice is subtly adapted, even during the relatively limited trial periods described here. Following an exploration by participants of MOVE’s tracking, mapping, range and speed and the associated learning, the two Tetsudo experts perform a version of their practice which concentrated on slow, passive movements, avoiding the cross-over of limbs, while facing forwards into the camera. A practitioner with the right level of experience in controlling their body movement adapts to the range of postures and speeds that MOVE can perform. This observation confirms Nehaniv’s assertion that even without a match in embodiment between ‘demonstrator’ and ‘imitator’, correspondence between the two can still be recognisable (Nehaniv and Dautenhahn 2002).

In the above sense, MOVE acts as limiter in a person’s movement range. While a performer could continue performing as if MOVE was not there, the experience is visibly much less rewarding, when they do. Then, MOVE moves when it should not or simply ends up in the ‘wrong’ place at the wrong time. When the performer understands the limitations, the experience does become rewarding. In this way, the architectural prototype and the performer have been mutually incorporated, to draw on a concept from social interaction (Fuchs and De Jaegher 2009). This interaction would allow an experienced practitioner a focus on the individual movements that they need to retrain or it might be fine-tuned to challenge a performer in specific ways.

7.5.2 A Selective Mirror

Given the lack of embodiment correspondence described above, the study demonstrated how MOVE acts as a three-dimensional, selective mirror for the person being tracked. It reflects the movements of a person back to them in a particularly structured way. This is relevant especially as mirroring and imitation are embedded in the movement arts (e.g. dance, martial arts). In dance, the interior architecture is often designed to support the practice by providing large mirrored surfaces in the dance studio. The mirror allows students to reflect on their own performance, while the performance of other students and the teacher also become visible. Beyond this, Architecture includes mirrors in other people-centred circumstances such as children’s play (ArchDaily 2012) and performance art (Kohlstedt 2004).

Digital mirroring also finds its use in dance and motion based analysis is now common in sports more generally, as Barris et al. review (Barris and Button 2008). The YouMove training system uses a half-silvered mirror and Kinect-tracking to measure to what extent people are following prescribed movement patterns and to help with retention (Anderson et al. 2013). In early work around dance, Hachimura et al. use a motion tracker to re-create a dancer’s movements through a 3D avatar (Hachimura et al. 2004). More recently, Kyan et al. deploy Kinect tracking within a VR projection to allow dancers to review their own performance with the help of a mapped avatar (Kyan et al. 2015).

Beyond physical mirrors in the environment and digital mirroring, there is an accompanying form of mirroring, both in dance and martial arts, in that students mirror and eventually imitate their teacher. Tetsudo can serve as a useful example here. It has a theoretical repertoire of movements that is taught by demonstration, mirroring and imitation. Each performer has an individual range within that theoretical repertoire and they express that range through their bodies in particular, performed instances. The movement ranges are not the same for two performers, even when comparing performers at the same ‘level’. In a typical Tetsudo class, the teacher will face a group of students, performing a series of movement routines, which the students will observe, mirror, learn and then be able to imitate without the teacher in front of them. Over time, the routines can become part of the students’ own movement repertoire. This involves the concentration and determination of the students and reflection on their own movements as the interviews and the video evidence have shown (see Fig. 7.6, P5 checking down).

In parts of the shared session between the two Tetsudo performers, it was then possible to observe the combination of the two forms of mirroring, outlined above: the environment mirroring one performer, and one performer mirroring another (via direct observation and via the environmental mirror) (compare section ‘Shared Session - Mirroring’). This is similar to the traditional dance theatre set-up apart from the fact that MOVE is of course not a mirror at all, as it breaks down the performed movements and reduces them to eight degrees of freedom that it can express. Some movements are simply not mirrored and what is mirrored therefore becomes amplified in the feedback loop. This selective mirroring and giving practitioners the choice over what is mirrored is a key property of MOVE that can be exploited in teaching, through the attenuation or amplification of what movement becomes mirrored, guided by the teacher or indeed the student. While this would be possible in screen-based (projected or head-mounted) alternatives, mirroring MOVE is physically highly immersive and performative. The Tetsudo performers agreed that a movement prototype like MOVE would be a useful tool for a martial arts beginner, as training is characterised by a focus on simpler, as of yet unconnected movement sequences. Breaking down and slowing down the movements to be learnt will actually help students in their learning.

7.5.3 Co-creation: Associating Human and Architectural Movements

Drawing on the broader context and previous work in this space, it is then possible to describe the embedding of movement-based interaction into the environment more specifically and how this leads to the co-creation of kinetic adaptive architecture. In Schnädelbach (2010) the general relationship between human behaviours and architectural behaviours has been sketched out. While making associations between human and architectural movements is principally included in this work, it was not its focus and was not described any further. Work with ExoBuilding (Schnädelbach et al. 2010) has lead to the description of the feedback loops that emerge between people’s behaviour and behaviours in adaptive architecture (including movement behaviours) and those remain at the core of architecture that is adaptive to people. Specifically focusing on movement, human movements as they are relevant for the mapping to architecture have then been described, including an overview of movement control and detection (Schnädelbach 2016). In this work, a number of examples from architectural history was used to begin to set out movement associations along the three axes of scale, expressiveness and control as well as the polyrhythmic relationships between human behaviours and environment as highlighted by Lefebvre (2013).

The MOVE platform was a direct response to the theme developing. It was specifically created to investigate making links between human and architectural movement. The study of the first prototype built with the platform has then delivered key insights to describe the co-creation process of movement-based interaction in Adaptive Architecture.

7.5.3.1 Creating Feedback Loop Through Correspondences—Opportunities and Constraints

When associating human movement with movements in the environment, the core concern of the architect or experience designer is to create the feedback loop between built environment and people. Creating this feedback loop involves making correspondences between the movement behaviours of people and the movement behaviours of the environment and this is evident in work across Adaptive Architecture such as Hubers (2004), Farahi Bouzanjani et al. (2013) and Bolbroe (2013). As we have seen in the study feedback, being selective about what becomes mapped to what is essential in this. What might correspondence here mean? The least correspondence would be a static environment, within the larger confines of environments never being completely static (refer to section Movement in Architecture). This would not present Adaptive Architecture in the sense that it is discussed here. The greatest correspondence would be a clone of a person (exactly the same embodiment) that somehow follows all movements of that person. This is clearly not possible but also does not concern Adaptive Architecture.

Within this extensive range of theoretical correspondence relationships, creators consider correspondences between environment and people with regards to spatial relationships, temporal relationships and control. Spatial relationships are concerned with mapping form, scale and degrees of freedom in movement between environment and people. Temporal relationships are concerned with mapping the tempo of movements between environment and people to create isorhythmic, polyrhythmic and arrhythmic relationships to use Lefebvre’s thinking (Lefebvre 2013). Finally, the architect needs to consider autonomy of the environment, i.e. can it move by itself or is it only reactive to the person inhabiting it, for example being able to lead a person rather than just follow it.

Creating anthropomorphic robots and therefore very close correspondences between people and environments was not the aim of this work, and when architecture is concerned it probably rarely will be. Also, as has been pointed out earlier above, people readily perceive correspondences, even if form and scale for example are very disparate between to items observed (Nehaniv and Dautenhahn 2002). In other words, architects and experience designers can achieve recognisable feedback loops between environments and people based only on a few points of correspondence. They have achieved this by creating a specific set of interaction opportunities and constraints (Fig. 7.10).

Fig. 7.10
figure 10

Developing Movement-based interactions within Adaptive Architecture. Architects configure correspondences between people and the environment to create feedback loops. Inhabitants inhabit those feedback loops to create egocentric form, performatively and expressively, in an embodied but distant and on-going fashion

Full size image

7.5.3.2 Creating Form and Feedback Loop

A person inhabiting the resultant Adaptive Architecture is faced by those opportunities and constraints to link their movements to those of the environment. In that sense, the inhabitant’s control over their environment is very much specified by someone else, for example describing exactly how particular body movements relate to particular environmental movements. The experimental work described here, has then demonstrated how people, through engaging with the feedback loop created by the designer, create form. The prototype gives people an unusual and on-going level of agency in shaping their environment: through body postures, architectural elements are arranged in space in a fluid and immediate way. This form creation process has a number of key properties:

Form creation occurs from an inhabitant perspective. It is created very much from the inside, from a central viewpoint in the environment to be created. In this, it appears most similar to design processes in Virtual Reality (Bourdot et al. 2010) allowing experts and non-experts to consider architectural designs in an immersive fashion. It might also be related to the perspective a crafts-person takes when measuring and laying out a room to be re-fitted, standing in the room during the design process. It is most dissimilar to the processes employed by architects, who view their designs mostly from the ‘outside’, and who produce detailed representations (e.g. drawings and models) before implementing a design. Related to the above, but not just considering the design process, the resulting architecture is fundamentally ‘egocentric’. Any form created by the tracked person, for example by aligning sets of panels up in particular ways, only ‘makes sense’ from their view point. Architecture is frequently concerned with framing views for inhabitants but the observer has to move into the correct position to appreciate this design. With MOVE, the inhabitant remains relatively stationary and arranges the environment around them to create form and to frame their view.

At the same time, this creation of form is very performative and presents an opportunity for expression. The interaction of the architect-inhabitant with their environment is clearly visible by others in the same space. In most observed cases during our studies, this form is not created for an audience and their point of view, and therefore the created form will be illegible, or at least it will appear very differently from a public viewpoint. A special case was observed when a second performer was deliberately added to the interaction, and MOVE was employed to re-enforce a threatening expression. This demonstrated how the expressiveness in the prototype could be used to amplify expressions or indeed to attenuate them. Finally, there is a peculiar absence of actual contact with the environment for something that looks and feels very much embodied to the tracked, interacting person. The coupling of body movement and panel movements is at a distance. This is very different from similarly constrained Adaptive Architecture (with regards to the range of overall movements it can produce), such as the aforementioned Schröder house (Kronenburg 2007, p. 26). It is also different from the slower process of creating form that a crafts-person employs, handling physical elements, assembling them in space, adjusting and fixing components where they belong. Most importantly, the form creation process is on-going, with the architectural form continuing to adapt, as long as the person is engaging with it. People are at the absolute core in the type of Adaptive Architecture described here.

7.6 Conclusion

The research presented in this chapter has synthesized previous work with the development and evaluation of the MOVE prototype. MOVE is a dedicated research platform to explore movement-based interaction in Adaptive Architecture. We have discussed the ways in which the feedback loop emerging in MOVE shapes and limits interaction and how selectively mirroring interaction can be employed. Drawing on the above, we have described the co-creation process that is enabled: (1) Architects and designers of motion-based interaction in Adaptive Architecture create the feedback loop for a specific application area, focusing on correspondences between people and the adaptive environment, (2) Inhabitants map their movements to architectural movements within the existing set of constraints to create form on an on-going basis.