Keywords

1 Introduction

Evaluation of interactive artwork faces new challenges in the digital age [1]. New methods for testing user experience and artistic qualities of interactive art are in high demand, while case studies and best practices contribute to the body of knowledge of the field. A promising direction in the evaluation of interactive digital art is to borrow and adapt theories and techniques from Human-Computer Interaction (HCI) [1, 2]. In this transfer, one possibility is a framework to identify and describe the dialectics and potential tensions between the visitors and the artifact, namely the Human-Artifact Model [3].

This paper presents the evaluation of an interactive sound and light installation that is explored by movement. The movement aspects of the installation have been previously reported in [4], and the sound aspects (especially the mapping between movement and sound, sound synthesis, and dynamic generation of control) in [5]. Here, we focus on the evaluation of the installation.

Previously, direct manipulation of three tangible interfaces equipped with sensors was controlling the sounds and colors of the installation [6]. With collaboration and embodied metaphors [7] in mind, the installation was designed as three tangible interfaces (fruits) mounted on branches coming from a central trunk. The original design was big; the fruits could only be grasped by both hands. Figure 1 depicts a smaller iteration with 5 blue fruits (leftmost), which can be moved with single hand. An evaluation of the original installation indicated issues on control: the participants did not seem to understand the underlying embodied and conceptual metaphors [7, 8].

Fig. 1.
figure 1

Different phases in the design refinement of a previous tangible installation. (Color figure online)

Full size image

More recently, we have implemented an indirection between motion and sound through a dynamic, intangible interface, movement qualities (MQ), and a virtual ecosystem (Fig. 2). A test session was conducted with 11 subjects, in a room at the university over a time period of 3 days. The subjects were asked to investigate the installation and to fill out a questionnaire afterwards. The gathered data was qualitatively evaluated, and a summary of results was reported [5]. However, an interpretation of the results under the human-artifact model remained unpublished.

Fig. 2.
figure 2

The Installation: interaction with light-rays.

Full size image

In both accounts, user associations indicated strong themes, such as “Very romantic; Hot summer night and bugs flying. Quiet space in a not polluted nature by water” or “I tried to play a wizard, waving his hands to shoot out fireballs”. There were also recurring references to play (especially children’s), magnetism, fantasy, and sci-fi. These themes, together with the history of the tangible artifacts depicted on Fig. 1 indicate an ecology of the installations. In this paper, we report on our preliminary work on the analysis of this ecology during interaction with the Installation. Our analysis is grounded on the Human-Artifact Model [3], and some of the findings will have direct impact on the design of a future work that will make use of specific gestures.

2 Background

2.1 The Human-Artifact Model

Human activity is mediated. Bødker and Klokmose developed and used the Human-Artifact Model [3] as means to analyze the use of artifacts within an ecology. The analytical scheme of the Human-Artifact Model combines analyses of human experiences and artifacts, and addresses the tensions between human skills and capacity on the one hand, and the action possibilities and affordances offered by the artifact on the other [9]. This is done on three levels reflecting the activity hierarchy: activity, action and operation. The activity level concerns why the user is motivated to interact, the action level with what the user does, and finally the operation level how the user performs the actions, together with the aspects and attributes of the artifact which support the corresponding level. The focus of an analysis with the Human-Artifact Model is on tensions between the experiences of a user and the assumptions of use embodied in the artifact on all three levels of activity.

Bødker and Klokmose [9] discuss how artifact ecologies evolve dynamically over time. Based on a study of iPhone users, they summarize this dynamism as three states that artifact ecologies iterate through: unsatisfactory, excited, and stable states. We note this approach as a future possibility to structure the continuity in history of the installation artifacts depicted on Figs. 1 and 2. Bødker and Klokmose also discuss [10] how the model can be used in design education as a conceptual and generative tool without going to deep into the Activity Theory behind the model. This is also our point of departure in this contribution. To our knowledge, the human-artifact model has not been used in design and evaluation of interactive art before.

2.2 Gestures, Movement Qualities, and Conceptual Metaphors

With the help of machine learning, interactive systems are able to learn and recognize gestures, which are subsequently used to control sound. A good overview of different techniques is provided by [11]. Machine learning can also be used to classify different intentions of a movement [12]. Instead of focusing on a complete gesture, the quality of the movement can be extracted. The Laban Movement Analysis (LMA), after the dance theorist and analyst Rudolf Laban, is a promising starting point in movement quality extraction. LMA is based on four main qualities: Body, Effort, Shape, and Space. Mentis and Johansson provide a good summary on the LMA, especially on the effort, and present an approach using movement analysis to control sound [13]. The authors of [13], however, consider the use of LMA as a too high level of classification for an untrained observer.

2.3 The Artifact Ecology: Interactive Multi-agent Systems

The term ecology is related to organisms within an environment, and how they interact with each other and their surroundings. An artifact ecology consists of multiple artifacts built for similar purposes but with slight variations and no clear delineation of when to use which artifact [9] –e.g. the installations on Figs. 1 and 2.

A software agent is an autonomous entity that observes and acts upon an environment and directs its activity towards achieving goals. Agents, for instance, enhance the musical performance possibilities: different approaches where some of musical parameters are controlled by agents in parallel to the user have been described in [14]. With the use of agents as procedures, the instrument takes on a life of its own and enhances the possibilities of the player. The installation Room #81 is a good example, where an agent is used to create both a soundscape and light changes to frame the users interaction [15]. Shacher and his colleagues recently suggested a classification of fundamental mapping relationships with the help of swarm simulations [16]. These include a number of strategies in making the mapping relationships less predictable and more organic. Other studies on agents and interactive sound include [17, 18]. However, none of these works facilitate the Human-Artifact model, to our knowledge.

3 Concept and Interaction Design

In our installation, we consider an audio-visual composition as a complex virtual ecosystem embedded in its enactive landscape (c.f., Fig. 2). The user, as a part of the enactive landscape is able to directly manipulate this ecosystem. The ecosystem is based on a flocking algorithm by Reynolds [19], which is represented in space through light-rays thrown by a projector. When the user enters the space of the ecosystem, he becomes a part of it, and the boids react towards him as a physical object. In parallel, the movement quality (MQ) is estimated in order to change the flocking behavior and therewith the way the user interacts with it. The details of the MQ estimation have been presented in [4].

Currently, the reaction of boids is considered as one of the three main conditions: hunted, passive and curious: hunted is a basic reaction towards fast and directed movements of the user. The boids start to organize in swarms, and seek distance to the user. Passive: when nothing of importance happens, the boids act independent (similar to a heard of wildebeests grazing). They move slowly, and seek distance to each other. The user has to move randomly and not to fast. Curious: when moving very slowly, the boids become interested in the user and come closer (similar to insects seeking a light source). These states seem to be well perceived with the help of audio-visual feedback. To create the aural feedback, the position and acceleration of each boid is forwarded to a Max/Msp patch using OSC. After updating the flocking system, the boids are rendered and projected, which closes the circle (see Fig. 2). Currently, all simulation and visual computing is done in C++ on openFrameworks, including the add-on openCV. The sound is generated in the environment of Max/MSP.

For simplicity, here we consider a case where the installation is placed in a room akin to a large digital musical instrument. This is to shorten the process of audience interactions with generic artworks [20]. Because of the intangibility of the light rays, however, the generated sound in the installation should be very directly coupled to the visual representation. The user should get an idea when a sound occurs and what is generating it. As a first design, we have considered three simplistic sound sources to experiment with the dimensionality of mapping, as suggested in [16]. Currently, not all of their parameters are used.

The tree conditions of swarm behavior outlined above serve as basic starting point in interaction design. While hunted was hinting towards a climax, the other two conditions are considered as calmer soundscapes. In the hunting part, the boids organize in flocks. A computer vision based evaluation was programmed estimate the flock formation, together and with their density and group velocity. Additionally, a temporal approach was implemented by the detection of collisions. Collisions happen from time to time and by distinction of different actors (boid/boid; boid/wall; boid/user) different sound qualities can be mapped towards them. So far, we have discussed the density coupling between three swarm behaviors and three sound sources. The first of these sound sources correspond to the swarms and implemented as a classical AM Synthesizer [21]. The other two correspond to collisions and implemented by physical models from the PeRColate library in Max/MSP: user/boid collusions trigger a marimba tone, whereas other collusions trigger a bowed bar tone.

4 Evaluation

A test session was conducted with 11 subjects, in a room at the university over a time period of 3 days. During this test session, the subjects were asked to investigate the installation and to fill out a questionnaire afterwards. In order to determine how consciously the subjects could control the installation, the subjects were asked to reproduce specific reactions. The gathered data was qualitatively evaluated. The three stages of the test procedure are described below.

  1. 1.

    Investigation of the installation: The subjects entered the room and were asked to explore the installation. They were left to explore freely, until they stopped by themselves or seemed to run out of ideas, repeating the same movements. The subjects were observed in terms of where they looked and which reactions the installation produced upon their actions. The observations were captured with a standardized form. This stage was planned to inform the operational level of the human-artifact model.

  2. 2.

    Questionnaire about the experience: After the investigation the test subjects were asked to fill out a questionnaire, which included personal questions (knowledge and background relevant to the evaluation), a question about how the different reactions were perceived, questions towards the sound and the connection towards to the light rays, and how the touch of the light rays was experienced, a question about the control experience, a question about the associations (to shift the focus towards the aesthetics), and questions regarding the installation’s sound aesthetics and visual appearance, plus personal engagement and general impression. Seven of the questions, especially those regarding aesthetics and interests, were posed with a Likert psychometric scale model, to provide a common measure. The subject was asked to rate within a range of 1 to 10; 1 standing for not at all, while 10 meant quite a lot/very. The Likert-scale answers were followed by a text-field with the request to comment. This stage was planned to inform the activity (why) level of the human-artifact model.

  3. 3.

    Performance session: The performance test was an experimental approach to see if the subjects were able to reproduce requested reactions. This was attempted, due to the fact that the vocabulary, which describes one’s experience, is often difficult to find. With the performance session, a specific reaction could be addressed and the result directly be observed. The requested tasks were designed after specific characteristics of the three conditions, hunted, passive and curious. This stage was planned to inform the action (what) level of the human-artifact model.

5 Results

Seven of the eleven participants reported some kind of technical background or have had some previous experience with interactive installations. Their age spanned 21 to 56 with an average age of 36 years. Three of the participants have heard about the installation before, while just one of them had some closer knowledge about the system. However, their results show no significant difference to the other participants. Their data are therefore evaluated equally with the others. The approximate duration of each session was about 30 min, most of the time taken by the questionnaire.

All qualitative data obtained has been coded by the second author, and complemented with his observations during the investigation and performance. The quantitative results are meant to complement the coding, by themselves they not reliable due to small sample size; nor it was the aim to gather statistics for interaction with an experiential installation. A qualitative evaluation of the results, was reported in [5]. The evaluation has highlighted the potential of the system to engage the user in all three levels of the human-artifact model.

5.1 Interpretation of the Results with the Human-Activity Model

The second author interpreted the answers provided by the test subjects according to the model described in Sect. 2.1. Sentences like, “I was looking forward to see how they would response” and “I would like to understand how it works and what it represents”, for instance, were interpreted as motivation for Exploration and Discovery. The pie diagram in Fig. 3, gives an overview over the occurrence of the ten categories extracted from the answers.

Fig. 3.
figure 3

Categories and factors in the evaluation.

Full size image

The two factors mentioned above seem to apply for all participants and tend to be the strength of the installation: Exploration, and Discovery. Other categories indicate orienteering aspects and qualities in addition to the motivation. Examples include Difficulty, Competition, Fantasy, Sensation, and Danger. Competition was considered for them who felt challenged by the autonomy the system provides and Fantasy by those who liked the openness of the abstract expression like mentioned in: “It was subtle but gives me imagination” or the fantasy of being a wizard. The possible tension mentioned by several subjects in understanding what to do is interpreted as Difficulty. The coding categorized mentions of musical actions or playing as a Sensation, which also inform the action level of the model. A rather unexpected result for the Danger example was: “… and I liked the sound that scared me at some point - that was funny”.

We are currently investigating how these categories interact with the performance and experience sections of the evaluation. Unlike any other utilitarian artifact, however, the primary engagement with our interactive installation seem to happen directly at the activity level, instead of the action (instrumental) level (c.f., [3, 9]).

6 Conclusions and Future Work

We presented an evaluation of a new interactive sound installation to be explored by movement via a synthetic flock of agents. In the installation, the relationship between gesture and sound are mediated by synthetic swarms of light rays. The system, as a running installation has been tested on 11 subjects. The results of the analysis by the human-artifact model [3] are encouraging, as the installation seems to provide many facets. The identification of Exploration and Discovery as main motivations is uplifting, considering our intention to create an installation allowing audience to explore experimental music. The evaluation highlights the potential of the system to engage users, as it seems pleasing to explore and discover different dynamic reactions. Finalizing the analysis with the model and its dynamics constitute our future work.