1 Introduction and background

Computing environments in the future will be populated by a rich and diverse set of devices and networks, many of them integrated with the physical landscape of space and artefacts. Since Weiser introduced the term “ubiquitous computing” and issued his call for computational technologies, being at the same time available and configurable for the user in her everyday environment and calmly fading into the background of attention, it has been clear that new modes of interaction and presence of interactive systems have to be sought for [1]. Early attempts to take the desktop metaphor of graphical interface design back to the real desktops and whiteboards by exploring new semantics of interaction was pioneered by Weiser’s group as well as by Buxton and others [2, 3, 4]. The idea to have a new and more complex set of physical handles to digital media promised an extended bandwidth in the interaction between people and technology, and, in line with Engelbart’s pioneering work on direct manipulation for graphical user interfaces, a new set of generic interface building blocks would open up a new realm for design of interaction technologies.

In parallel to the work of Weiser, Wellner and colleagues argued for a new and broader interpretation of augmented reality, turning computational augmentation into an enhancement of practices well established with the interaction of more mundane artefacts [5]. Fuelled by ethnographical studies of work, researchers such as Mackay et al. [6] suggested augmented environments, where computational resources were brought into play as extensions of, for example, the paper flight strips that traffic controllers used to control airplanes as they pass through different traffic sectors. Such an approach is not in opposition to the development of new interaction modalities but it shifts the balance from generic interaction scheme to situated embodiment of interactional possibilities. Ishii and Ullmer [7] forged these two approaches into a wider program for tangible interaction. With the ambition to create seamless interfaces between “people, bits and atoms,” Ishii and Ullmer have expanded the new field of design to include an integrated reshaping of desks, boards and rooms.

The growing number of experimental ubicom installations has helped to shift the attention of interactive systems away from individual work settings and towards larger collaborative environments that were traditionally the realm of other designers. After some years, during which automatically generated context information created high hopes for how computational technologies could be made to match the complexity of user behaviour [8], we are increasingly seeing suggestions such as HP’s Cooltown project for open infrastructures and end-user configurable systems, which may have a lower intensity of computational monitoring, but, on the other hand, appear more easily extendable to widespread real-life settings [9].

This new type of extendable systems with open boundaries provides traditional HCI research with important new challenges [10, 11]. In many environments, the landscape is evolving and changing with the tasks at hand, and users will most likely make use of functions and tools in unexpected ways and expect to be supported in doing so. Newman et al. [12] argue “that systems should inherently support the ability of users to assemble available resources to accomplish their tasks. In a world of richly embedded and inter-connectable technologies, there will always be particular combinations of functionality for which no application has been expressly written” [12]. This view reflects our experiences within the IST Project AtelierFootnote 1, which develops a set of architectures and technologies in support of inspirational learning in two areas—architecture and interaction design. We observed how students configured and reconfigured their workspace, as well as the relationships between design representations. This motivated a design approach that focuses on configurability as an important feature of mixed-media environments.

The article starts with an analysis of observed practices of configuring. It then presents three design examples that highlight different aspects and levels of end-user configuration. The discussion part summarises the diversity of configurations, and investigates in particular how, in augmented or tangible computing environments, configurations of digital media extend into the physical setting and into the qualities of the objects created.

2 How design students work

For architects, configurability is connected to the properties of a space. Flexibility connotes the possibility of relatively simple changes to a design so as to adapt it to different or shifting social uses (e.g. moveable walls). Variability means that a designed space or artefact, without elaborate transformations, can accommodate a variety of functions [13]. The backstage and the garage stand for spaces in which everything is possible. But there are also some quite elaborate examples of configurability, such as a building under construction by Diller and Scofidio, which has been conceptualised as “a fundamentally updateable, technologically and profoundly rearrangeable (physically)” building setup. The architects used the metaphor of open-source code for modelling the building as a space “capable of being rewritten, upgraded, reprogrammed, reconfigured to accomplish previously unanticipated tasks” [14]. The building is to be interactive, not only in the exhibition and informative parts, but also architecturally. A visitor, entering the building physically or online, can manipulate parts of the façade. “Smart walls” made from liquid crystals between conductive film and glass panes allow for an electric current to regulate the level of transparency and daylight conditions within the building.

Configuring and reconfiguring, although with much more mundane means, is also part of students’ practice. Students voice a strong need to adapt their workspace so that they can exhibit, perform, engage in group work or work alone, build models, have a nap, make coffee, interact with material and odd objects, etc. At the beginning of a project, the architecture students set up their workspaces. As the project progresses, they become dense with design material, which is exhibited on the surrounding walls and on parts of the desk space. Sketches, plans, models, a panorama print of a site and the computer are all assembled in one desk space (Fig. 1). One student has put two desks on top of each other to make room for a desktop computer, turning the desk into a three-dimensional space. Here, configuring spatial elements and tools is very different from the predesigned mobile and flexible “individual workstations” that have become part of office design [15]. These are highly personalised workspaces, whose features and components grow over time, expressing students’ identity as well as the progress of their work.

Fig. 1
figure 1

Workspace of a diploma student

Full size image

The concept of configuring can also be applied to the ways in which students arrange and rearrange design materials. Project work proceeds through developing a large number of design representations. These are characterised by the expressive use of materials and are sometimes animated and presented dynamically. As each of these representations exhibits and clarifies particular aspects of the design, it is important to forge and maintain connections between them. In many instances, students configure and reconfigure design materials so as to read and reread the configuration from different points of view, and to be able to go back to a moment where a particular issue emerged. In the process of conceptualising and detailing, the design representations and their relationships change continuously. Arranging and rearranging material in the workspace is an essential part of this process, with the physical landscape of things on the walls and tables being in constant movement. This resonated with earlier findings on work practices in (landscape) architecture, where we argued that: “What emerges is that manipulating the presence and absence of materials and bringing them into dynamic spatial relations in which they can confront each other are not just a context or prerequisite for doing the work; rather, they are an integral part of accomplishing the work itself. To manipulate the context is to do the work. Typically, what is important is not just to create or change a document or other materials, but to do so in the presence of and in relation to others” [16].

Configuring as a practice is intricately linked to the fact that, in evolving environments, such as the architecture class or the interaction design studio, the boundaries of activities are continually moving [17]. Our observations helped identify two meanings of configurability:

  • Adapting a space to a diversity of uses and identities—which is achieved through e.g. appropriating a space, personalising it and configuring it in support of diverse arrangements, such as solitary work, group discussions, performing and presenting, building models

  • Configurations of artefacts within the physical space—with artefacts changing their position in relation to others and different configurations being expressive of conceptual, chronological or narrative links between them

Embedding digital media in physical environments cannot be simply understood as an extension of practices we observed in the physical world. Things need to be designed so as to support “the ability to improvisationally combine computational resources and devices in serendipitous ways” [12]. The following chapters examine several experiments with tangible augmentation of student design studios with the intent to better understand the requirements of end-user configuration in mixed-media environments.

3 Mixing media in the design studio

The first widely published examples of tangibly augmented environments made it clear that such environments are a much more complex mixed-media setting than the preceding individual workstations [18, 19]. Hereby, they also made it evident that a new set of building blocks has to be provided to designers in order to let such environments flourish. Myers et al. [20] argue in a review of past and present toolkits that we probably have to operate in a very open field for many years to come. One of the obvious needs identifiable today is, however, that the design material for leading edge augmented environments have to integrate hardware and software in ways that reflects the fact that the digital and the physical domains are no longer separated in these environments [20]. Greenberg and Chester’s suggestion for a widget-like toolkit of Phidgets [21], giving designers building blocks to control and combine the operation of basic interaction elements like contactors, servomotors etc., has, in student trials, resulted in interesting new examples of mixed-media interfaces. In a slightly different direction, Ballagas et al. [22] have developed a kit of generalised tangible interaction devices for an intelligent room, making it possible to easily design more aggregate interaction systems. Strommen [23] has suggested yet another type of toolkit, linking computational agent behaviour to an actual embodiment of doll-like physical artefacts that can be freely placed in the environment of use. Of particular interest to us is the work on embodiment of electronic tags pioneered by Want et al. [24] that embodies basic computational functionalities in border objects that can operate both as handles and as actual media in settings rich in both physical and digital media.

The problem with most attempts to provide designers with the components from which mixed-media environments can be made, however, is that they still appear to be products of engineering labs. As pointed out, for example, by Djajadiningrat et al. [25], tangibility may be new to computer engineers, but it is certainly a well-known challenge to product designers. Designers will need toolkits that open up rather than seal off the embodiment of interaction components. Furthermore, other voices from the designer community claim that even basic computational functionalities have to be made accessible and mouldable for exploration of the interaction designer [26]. There may be different ways to pursue the exploration of form and function of interaction elements. Hutchinson et al. [27] report on a playful and experimental strategy of probing for new interaction experiences in the home setting by embodying basic technological possibilities in ways that can be grasped by collaborating families.

Although inspiring, this approach has the limitation that the embodiment tends to become too casual and ad hoc to form the basis for a genuine genre of interaction design relevant, for example, in design education. In the Atelier project, we gave design students the opportunity to explore and appropriate new combinations and linkages between digital and non-digital media in on-going projects. As in the work of Want et al. [24], we supplied them with possibilities to tie digital media to physical objects through the use of RFID tags. Similarly, we provided them with simple applications for associating barcodes or particular manipulations of touch sensors to computational resources. Sensors, tags and barcodes provided a simple way to associate media files with the environment of space and objects. Associations of physical input, digital media and output could be edited and loaded through configuration tables that are stored in a database. Associating a physical input to digital media can result in a diversity of configurations. For example, the input component attached to a barcode scanner can be configured to read one barcode as input or a series of barcodes. Moreover, the media output can be configured to display one or more digital files on one or more projection surfaces in the space.

3.1 Mixed-media spaces: sensors, tags and projections

Trials were organised at two sites: at the Master Class of Architecture, Academy of Fine Arts in Vienna, and the Interaction Design Studio, Malmö School of Art and Communication. At each site, groups of students (in each trial between 10 and 20 students in small groups) were carrying out their practical projects in the Atelier environment. The trials lasted between two and four weeks. The results of the first cycle of trials were different configurations of physical inputs and digital outputs, and a variety of projection setups. Students came up with a wide range of ideas of how to integrate interactivity in physical objects. They used the space as a resource, e.g. by recreating elements of remote places in the studio. Multiple projection surfaces were arranged through “bricolage” in the space. Students used barcodes for creating associations between media files and parts of a physical model of a building. In Fig. 2 (left), the site represented by the model is a large garden with fruit trees and an old house. The model was animated with sound and projected images of interiors, of the old house and of different perspectives onto the surrounding garden. Touch sensors were also integrated in parts of the model (Fig. 2 right) to create interactive models or navigation artefacts for presentations.

Fig. 2
figure 2

a Barcodes placed onto models. b Touch sensors integrated into model

Full size image

In preparing a project presentation, one of the architecture students plotted out her CAD plans with barcodes on them. In one of her printouts, she had integrated the barcodes into a diagrammatic representation (Fig. 3 right corner). She presented her work using multiple interactive artefacts that triggered the playing of sound and visual media on a projected screen (Fig. 3 upper right). Barcodes were integrated into posters, which displayed plans and diagrams (Fig. 3 upper left). A physical model of the section of the stadium and its surrounding environment was made interactive with touch sensors (Fig. 3 bottom).

Fig. 3
figure 3

Configurations of sensors, artefacts, digital media and projections

Full size image

3.2 Configuring spatial elements

The use of multiple projections invited students to use the space as a resource. Students arranged their own projection setups, recreating aspects of a remote place, by, for example, using projection screens and hanging posters in the shape of this place. For example, a student projected images of two residential buildings with two back-projections, which he arranged facing each other. He visualised the transformation of the balconies into seating arrangements for viewing a soccer game in the space in-between, performing these changes while the class was watching. The detailed plans of his interventions in the buildings were projected onto the space between them (Fig. 4).

Fig. 4
figure 4

Creating spatial elements with projections

Full size image

One of our interventions in the physical space was a grid in support of configuring the workspace for different activities (Fig. 5). The grid facilitated isolating smaller partitions of the room to be used for smaller groups and installing diverse projection setups. Most importantly, it supported students in furnishing their project spaces in whatever way they wanted, and in rearranging them whenever activities changed. These arrangements could be performed in a varied topography in the space, with the possibility to experience things from above or below.

Fig. 5
figure 5

Physical infrastructure for projection spaces

Full size image

3.3 A new multimodality in design representations

The design students’ experiments revealed to us the importance of supporting the spatiality of configurations. Students created immersive installations by arranging projections spatially and they introduced a new multimodality in their representations where different media were explicitly connected: the physical model through touch sensors, the barcodes printed in posters, tags in cards, all were connected with pictures, sounds and video on multiple projectors. New hybrid forms of design representations emerged in the work of the students; e.g. mixing issues of scale by shifting between plan drawings and full-scale imagery. The possibilities to mix static and dynamic representations were also explored in projections of live images on scale models.

Tagging physical models or drawings with barcodes associated to digital media was particularly frequent. This bears similarities to the use of barcodes on paper cards to activate digital presentations, as suggested with the paper palette [28]. But unlike the paper palette, the students did not develop their use of barcodes for a preformatted type of presentation like a slide show. Instead, they integrated barcodes as a graphical element in their drawings and plans, and adopted stage-like practices of projecting images, often onto non-standardised surfaces (physical objects, customised projection screens), in order to convey particular immersive perspectives on their project.

Both physical and digital materials were collected and explored during the students’ design work in a variety of ways. Digital images collected on field trips were used for projection on various mockup materials, and images could be used for both establishing a context or for capturing particular interesting textures. In this way, new constellations of physical and digital material were concurrently sought for involving relinking and retagging materials. In the first trial, this was poorly supported by the technology we provided to the students, as they were not able to make such reconfigurations on their own.

In general, the experiments indicated that design work in mixed-media environments provided the students with a new and fertile setting for exploring truly mixed-media representations where the most promising results were obtained when digital and non-digital materials were forged together into new hybrid objects, often with dynamic behaviour. The use of tags and sensors to invoke digital media meant that existing practices of sketching and modelling with physical materials were enhanced without having the students to switch to GUI-type interfaces.

While the experiments with our students were promising, it also became clear during the evaluation sessions that the students found it difficult to work with the environment we provided. They complained about the complexity of coupling digital and physical configurations. They also found it problematic that several weeks of work remained distributed in the environment and were not easily portable outside the environment, and could not be documented properly in their portfolio of projects.

In a second round of trials, we addressed these problems by developing prototypes that provided basic formats for collecting, configuring and documenting mixed-media representations. We made available more complex applications to mix physical and digital objects, physical infrastructures supporting configurations of space and configurable furniture. The next section describes the “Tangible Archive”; configurable furniture we used in the trials of the second cycle. The section entitled “Mixed object table” describes more complex applications we developed to mix physical and digital objects.

4 A tangible archive

When designers work, they collect inspirational materials as different media. This collection of materials forms an important resource base for much of the design work. As we have already mentioned, designing can be seen as a kind of bricolage where different materials are brought together to explore and envision design possibilities. Some of the authors have, in a previous project, written about the problems in organising digital materials in ways that facilitate a designerly exploration, and have suggested the “Wunderkammer” as a metaphor for such collections [29]. The problem encountered when working with mixed-media representations is even more urgent if the availability of collected materials cannot be made tangible in the design environment. In the initial trials with design students, we provided them with the tagging of selected design materials, but as already mentioned, the collection and storing of these materials became a major obstacle as it depended on our support. Furthermore, some of the most promising experiments made by the students involved setting up and exploring dynamic immersive environments for their design representations in a scale of 1:1. However, the students found it tedious and often too difficult to sketch early ideas in this way.

To overcome this problem, we developed, together with a group of master students, what we called the Tangible Archive. The Tangible Archive can be seen as a configurable working area where designers can store, organise and manipulate design material in different media gathered for a particular project or set of projects. The basic principle is, as in the earlier experiments, to use the tagging of physical objects as a way to connect to digital material and computational resource, but in the archive with an emphasis on providing components and configurable component assemblies that embody more general affordances and interaction schemes. As in Grönbæk et al. [30], we search for components, which can form the glue in mixed-media objects and extend our conception of hypermedia as well as provide a basis for new configurations of the design environment. Our goal was, however, not to accomplish new types of augmentation that can provide a new tooling of conventional design practices as demonstrated, for example, in the Workspace project [31]. For us, the Tangible Archive is not meant to be a fixed setup but, rather, a collection of design patterns from which the design students can gradually transform their collected design materials into exploratory full-scale mockups.

In addition to the work previously cited on tagging, we were inspired by the mediaBlocks project by Ullmer et al. [32], where an environment for manipulating digital video can be prototyped by a combination of container-like media blocks and appliance-like manipulators. But unlike Ullmer et al. [32], we did not want to target a particular kind of media manipulation. On the one hand, we wanted to provide a very open frame for the exploration of design materials. On the other hand, we attempted to provide the starting point also for interaction semantics, somewhat in the direction of the concept of DataTiles suggested by Rekimoto et al. [33]. The DataTiles provide a basic set of interaction semantics combining the (designable) physical form of tiles with an (exemplary) systemic scheme for how tiles can be combined to form more complex interaction patterns [33]. We did not take it as far in terms of emergent functionality, but wanted also to address issues of physical embodiment. In the first version of the Tangible Archive developed by a group of master students, a bookshelf with trays could be filled with physical objects tagged with RFID tags. One or more digital files were associated to each tagged object, and interaction points equipped with tag readers made it possible to display or print digital material associated to the tagged object placed on the tag reader. What commands were executed depended on special tagged command cards placed at an additional tag reader. The command cards also made it possible to copy and delete digital material from the tagged objects. Finally, a digital camera was associated with the archive to enable users to add new digital material to tagged objects.

Compared to earlier student experiments, we found that this group, in a more generic sense, had been able to take advantage of affordances of the physical environment, like organising tagged objects in particular trays. They also made concrete embodiments of mechanisms for connecting, displaying and copying digital materials associated to physical objects.

In our research group, we developed the concept further to see if we could use the tangible archive both as a family of design patterns and as a concrete toolkit for other design students to work with. We saw the tangible archive as consisting of an overall pattern for keeping and organising collected material that may eventually be turned into mockups of a final design. We suggested to the students to start their work by establishing this pattern as an actual literal storage for the materials they collect on field trips etc. Then, on a more fine-grained level, the archive consists of other patterns that we labelled zones. A zone is an area in the tangible archive which combines a set of input and output devices, such as tag readers and projectors, in order to accommodate certain activities, such as entering or organising material. Finally, we developed the first steps towards a collection of interaction semantics for individual interaction points that could also form the starting point for design patterns.

Practically, the Tangible Archive consists of physical objects such as models, material samples, pictures or “objets trouvés” and digital files organised in a hypermedia database, either as individual files or hypermedia documents or projects. The physical objects are linked to the digital files by either barcodes or RFID tags. The basic principle is that every digital file is associated to a physical object. The Tangible Archive has, as a minimum, an organising zone, where digital media associated to the physical objects can be displayed and copied, and an entrance zone, where new digital materials can be entered and linked to physical objects. The archive frame (Fig. 6) is built up of modules of size 48×48 cm made of either plywood, transparent Plexiglas or semi-transparent Plexiglas (for projection), all 4 mm in width. The modules can be combined to form cubes, shelves and vertical or horizontal working or projection areas, in a scale appropriate for working in scale 1:1, according to the needs in the project. Modules are joined together manually by ready-made joints.

Fig. 6
figure 6

The archive frame populated with a diversity of physical objects

Full size image

By supplying the students with such a practical building kit, we wanted to ease the process of transforming the initial archive into relevant stages for exploring and presenting design ideas. For the zones we prepared, special plates with holes for placing input and output devices and the individual interaction points could be moulded on top of a set of basic building elements, consisting of small tagged objects (to be connected to other objects) and matrices for the plate holes to signal which objects could go where. The underlying technology, including the infrastructure and hypermedia database, was the same as in previous experiments.

4.1 Basic interaction principles—tags and tag readers

In the archive, digital media associated to physical objects can be displayed and manipulated by moving the tag on the physical object to a tag reader. The resulting action depends on the configuration of the tag reader. In the archive, all RFID tag readers are placed stationary in the archive to form preconfigured action spots, whereas barcode readers are used when digital media must be invoked where the barcodes are placed.

Physical objects tagged with RFID tags are, in the archive, associated to what we call a carousel of digital media (corresponding to a hypermedia document optionally consisting of more than one file). The action associated to an RFID tag reader will be active, as long as the RFID tag is placed on the tag reader. Physical objects (such as print outs with thumbnail pictures and barcodes) tagged with barcodes are, in the archive, only associated to one digital file (corresponding to an object in the hypermedia database). The action associated to the barcode will be active until a new barcode is read by the barcode reader.

4.2 Entering new materials in the archive

There are two types of entrances to the Tangible Archive that makes it possible to bring in new digital media (new digital media may also be generated in the archive by manipulating other media, but this is not covered here).

The drop entrance is a dedicated work zone, where media from USB-enabled devices, such as digital cameras, can be entered by connecting the device and placing an RFID-tagged object on an associated tag reader. The device is treated as an external hard drive and the files on the drive are stored in the hypermedia database and associated to the tagged object as a carousel of files. These may later be copied (or deleted) and manipulated in other zones.

The email entrance is a dedicated work zone made tangible by a printer printing barcodes (possibly also with thumbnail icons). E-mails with attachments e-mailed to a particular address will be parsed by the e-mail entrance service so that individual media files (possibly with meta information) are associated to individual barcodes. The files may later be copied (or deleted) and manipulated in other zones.

4.3 The configurator deck of cards

The archive may be used by several people at the same time. A part of the archive reserved for one particular activity is called a zone. The simplest zone consists of two tag readers and a display. One tag reader identifies which media should be displayed/manipulated, and the other tag reader offers the possibility to copy what is displayed to the tag placed on this tag reader. There may be a number of different ways to display media. They can be of different forms, like visual or audio. Media in, e.g. a carousel, may be displayed sequentially by placing and lifting the tag from the tag reader, or superimposed, by displaying all media in the carousel or from different carousels simultaneously. In order for the people using the archive to be able to configure it according to their needs, a configurator deck of cards is provided for each possible type of work zone. The deck of cards has an individual card for all the components that can be combined (Fig. 7). The card does not only represent a specific piece of hardware (like a tag reader) but also an associated service (like display sequentially). The cards are graphically designed so that, when placed on or close to the physical component, they give a direct indication of the configuration. Each deck of cards also has an overview card briefly stating the function and interrelations between components. At a later state, the configurator deck of cards can be used for dynamic configuration of the archive. Barcodes on the cards and on the hardware can be subsequently read by a barcode reader to invoke the configuration.

Fig. 7
figure 7

Tag reader integrated in the furniture (left); building different forms of furniture with modules (upper right); the configurator deck of cards (lower right)

Full size image

4.4 From archive to mockup

Design students worked with the tangible archive in a two-week design workshop. During the workshop, the students worked in groups to explore what we called semi-public places (a public library, a playground in the city gardens and cafés on a public square) and they were asked to design an interactive installation that conveyed what they found to be interesting qualities of the places they had studied.

They made video, audio and still-image recordings of the places they visited and they collected physical items from the area. After an introduction to the tangible archive, they built a first version of the archive for the collected material.

The students used the archive frame to set the scene for the exploration of their material. The group working with playgrounds made a table-like archive where collected digital materials were connected to tagged leaves gathered in a heap in the middle of the table. Images and videos could be displayed on a sphere mounted above the heap and people were supposed to sit on the floor in a circle around the heap. The group working with cafés created a closed cinema-like space where one could wander along the cafés. The group working with the library built a counter-like structure using barcodes and barcode readers in ways similar to the way library staff register books.

As the groups continued their work, they experimented with both the physical frame and the interaction zones to try out ideas. One group was particularly interested in the e-mail entrance and the possibilities of using this in connection with a public billboard for buying and selling stuff between people in the city. Like in the original version of the tangible archive, they made it up to the people actually present in front of the billboard to pick up announcements e-mailed to the board from the printer and put it on the board. The announcements were printed out at the board with a barcode connected to additional contact information. Another group originally working with the heap of leaves made a variation of a display zone where two people placed at some distance to one another had to collaborate to put together four pieces of an image when each of them only controlled two of the puzzle pieces.

In general, we found that the tangible archive enabled the students to work in quick iterations with full-scale mockups of interactive installations. The students tended to make smaller variations of the zones we had provided for them. They did not develop conceptually new zones (even though we encouraged them to do so), and, even with the setups we had demonstrated to them, they tended to work further with those that they already had a familiarity with. For example, we found that the students preferred working with barcodes and barcode readers rather than with RFID tags. Even if this, to some extent, may be explained by the fact that the barcode technology in our setting was more reliable than the RFID tag readers, we found that the RFID tag readers and tags also in the students’ view were more difficult to embody in their design without considerable efforts to develop appropriate interaction schemes.

5 Mixed objects table

The Tangible Archive and its exploration in the student trials addressed the design students’ practice of working with mockups in the scale of 1:1. The initial trials also showed a considerable interest among the students in working with mixed media in connection with tabletop-scale models. Already in the first trials, the architecture students used digital imagery to capture and work with a visual texturing of physical 3D models. For the second round of trials, we wanted to facilitate this way of working further by providing a more elaborated setting for work with tabletop models. Others have been working with the augmentation of tabletop models. For example, Underkoffler and Ishii [34] have suggested a luminous-tangible workbench where calculations based on visual tracking of the physical model is made tangible by the projection of shades indicating, for example, noise levels or sun shade in the surrounding of the modelled buildings. Schmudlach et al. [35] have been working with similar setups where digital technology is used to keep alignment between a digital and a physical 3D model [35]. Our purpose was, however, not so much to invoke computational resources for calculation or to relate to digital models, but rather, to enhance the manipulation of the direct visual appearance of the physical 3D models. Work in this direction has been carried out by Bandyopadhyay et al. [36] and by Raskar et al. [37], emphasising the visual possibilities in overlaying physical models with projected digital imagery. Their work does not, however, in the same way as ours, take its starting point in the way design students normally work with tabletop-scale models. For our purposes, we developed the Mixed Objects Table. It is a table with a semi-transparent glass tabletop, which is a projection screen. A mirror is mounted underneath, making a back-projection system. Users can combine the table with different projection setups. In the Atelier space at the Academy, for example, several movable large-size projection screens are available. Those screens can enhance the table projection. Moreover, the Mixed Objects Table can be coupled with the Texture Brush. The Texture Brush is a tool which makes it possible to “paint” objects, such as models or parts of the physical space, applying textures, images or video, scaling and rotating them. The Texture Brush uses a real brush, but a virtual “paint.” Finally, ARToolKit [38] optical markers can be used in the Mixed Objects Table environment.

5.1 Table and projection setup

The table itself is a movable (it is equipped with wheels) piece of furniture with an integrated mirror. Two tag readers and sockets for USB devices, e.g. for web cams, a scanner or a printer, have been integrated into its frame. The table can be easily adapted to different uses. The simplest one is playing media by projecting them via a mirror onto the table. Projecting the desktop turns the table into a workspace, with the peripherals being directly attached to it. Furthermore, it can be personalised in a way which resembles Naoto Fukasawa’s design “personal skies,” which has “two elements, a chair that adapts chameleon-like to the clothing of the user, and a means of personalizing the work environment by projecting a personal ceiling above the desk (it could be an image of the sky in a choice of season or weather conditions, or of the home), sending a customized message to the rest of the office like a screen saver” [15]. At the Academy, students use the table for experimenting with objects, such as scale models, in a projected environment (Fig. 8). More complex setups include additional projection screens. Surrounding the table with various projection screens hanging on the grid enhances the simple projection setup, and invites the students to experiment further. The most complex setup explored so far is using the table with surrounding projections and a Texture Brush simultaneously.

Fig. 8
figure 8

The mixed objects table (left) projecting a CAD plan onto the surface of the table. Two tag readers and one of the USB slots with attached webcam (right)

Full size image

5.2 Texture brush run-time configurations of textures

Students can position their physical models on the table, e.g. on top of a projected plan, and use the Texture Brush for “painting” the model using various textures. The Texture Brush uses a real brush, which is tracked by an infrared camera. The brush is equipped with a simple button as well. The system projects the virtual “paint” only where the brush passes by and the button is pressed. In this way, the user has an impression of real painting. The architecture students rarely use simple colours for painting, but apply coloured textures, snapshots or even videos to their models. The Texture Brush is an application that creates mixed objects [39], where integration of the physical and the digital happens within one single object. This notion goes beyond simply enriching a physical artefact by linking it with content in different media. In the case of the Texture Brush, the link is such that the properties of the artefact itself can be changed, by applying colour, inserting movement and context, and varying its dimension in relation to other objects in the physical space.

The students can configure the Texture Brush in many ways. They can manipulate the brush size and shape by selecting these attributes from a menu bar, located at the bottom of the projection area, which is constantly displayed (Fig. 9). Working tools like “polygon fill” that are known from applications like Adobe Photoshop, have been integrated in the Texture Brush application. This allows the students to work with the Texture Brush much in the same way they are used to working with applications they know. They can choose from a number of textures to use, including animated textures (videos), and the projection of the textures can be moved, scaled and rotated. The whole interaction is done using the brush as only input device.

Tag readers and RFID tags and barcodes can be used to load the textures into the system at run time. Any image or video stored in the hypermedia database can be used as a texture to paint the objects with. The texture brush can be used to paint only one side of the model. If the user wants to paint the model from more sides, more texture brush systems have to be used simultaneously. Some of our students used two systems to paint the models from the front side and from the top. There is also an element of physical configurability, with the possibility of placing physical objects on the table and varying the background, using projection screens and varying the “ground” on which the objects are placed by using the back projection of the table.

Fig. 9
figure 9

Configuring the Texture Brush for painting a model (left), on the border of the table is the menu bar. A painted model from two directions with background (right)

Full size image

5.3 Optical markers: configuring the virtual space around an object

Besides simple projections on the table (Fig. 8 left), the space around a model can be further enriched using ARToolkit optical markers. Optical markers are simple printouts which can be placed on the table. A web cam is used to capture the physical model and the markers. The position and orientation of the marker relative to the camera can be estimated. Every marker represents a 3D object (such as a tree in Fig. 10), and when the position and orientation of the marker is known, the object corresponding to the marker is added to the image. Students can place the real model on the table, paint it with the real brush and virtual paint, place a few real markers (paper cards), project the ground-level image onto the table and capture the whole setup with a web cam. What we get, in real-time projected on a display, is the movie of the composed scene with the 3D objects popping out of the markers. In this way, we have a real object standing on the real table, but with the virtual textures on the object (which have been painted using a real brush). Furthermore, we have a virtual playground for our object, real markers, which do not make much sense in the real world, and virtual 3D objects corresponding to the markers, visible only in the real-time movie. The wall on the back side of this arrangement may be simultaneously used for creating contexts against which the model can be viewed. It is part of architects’ practice to take pictures of these experimental setups and to merge them with the other design material.

Fig. 10
figure 10

Using optical markers for inserting a virtual tree into a physical model

Full size image

5.4 The “configuration poster”

Configuration of the space with such a variety of projection possibilities is not trivial. Students have to configure the system so that specific images are projected onto the specific projection screen, or used as a texture in the Texture Brush or as a tabletop image. We have designed a simple physical handle to configure the space. The configuration poster uses barcodes to specify the receiver of a texture or any other input. This approach is similar to the solution in the Tangible Archive example, but instead of using a dedicated tag reader and a set of command tags, a poster displaying the possible connections between inputs and outputs using barcodes can be used to configure the system. There is a barcode for each command. In this way, the tag reader and the barcode reader input can be configured to, for example, display the media file associated to a specific tag or barcode on either the desktop or to use it as a texture with the Texture Brush (Fig. 8b). Additional barcodes have been added to specify printers and projections on the cave corner as other output components. Also, other tag readers can be connected to other outputs—for example, one tag reader is connected to the Texture Brush and the other is connected to the desktop projection. These connections between input and output persist as long as they are not reconfigured by the students. The configuration and reconfiguration can be performed at any time, dynamically changing the setup of the workspace, using only configuration poster, barcode reader and barcodes that correspond to the images stored in the database. This allows students to adapt the setup to their current needs, while they are working, making the configurability part of their work with the setup. As the Mixed Objects Table uses the same hardware as the Tangible Archive (tag reader, display, etc.), with similar functionality being available, students can choose the space configuration—table or archive—that suits their current working situation best.

5.5 Towards new modelling practices

The Mixed Object Table provided new ways of mixing digital and physical objects. Students used the table and surrounding projection screens as a stage for their models. Nine groups of two students used the Atelier environment to experiment with virtual textures and backgrounds for white models of buildings they built. Nearly 1,000 digital pictures and videos were entered into the database and were used to paint the white models and background. The groups created several configurations of textures and backgrounds for each model, some of which were conceptual, others very realistic, like a video of moving water for a pool (Fig. 9 right). The students often were using the Texture Brush from two different directions running two separate applications (Fig. 9 right). After having painted and experimented for several hours, the student took digital pictures inside and outside the “painted” models from different directions. These were then projected on large screens during final presentations, creating immersive spaces with multiple projections. The students were enthusiastic about the texture brush but mentioned several difficulties in the interviews. They lamented that they needed to adjust projectors and surfaces every time they started working again after another group (tape was used to mark position and height of projectors). As each model was of different size and was painted from different directions, each group had to rearrange the table and projectors into different positions (and also the parameters on the projectors were changed due to the difference in the size of the picture). This resulted in the impossibility for some groups to restore the sessions, juxtaposing exactly the painted texture and the model. In the presentations, some groups had to paint the model from the start, and some could show only the pictures they took.

6 Discussion

We began with a notion of configurability, which was shaped both by the architectural concept of adaptability of a space to a diversity of uses and identities, as well as by our observations of “dynamic spatial relations of design materials” as an important aspect of design practice. The three examples we provided explore different aspects of configurability of a mixed-media environment: associations of inputs, media and outputs; spatiality and integration with artefacts; configuring furniture and work zones (Tangible Archive); and real-time configuration of mixed objects (Mixed Objects Table).

Our first example shows how, using physical interfaces such as sensors, tags, barcodes and projection setups, a space can be configured to form mixed-media stages. Physical interfaces are integrated in diagrams and physical models. Projection setups, exposed in the space, create spatial elements that provide a stage for enacting scenarios, performing presentations or travelling through media material. In contrast to a dedicated meeting setting with one projector and one projection surface, our experimentations reveal ways of using the space as a resource for varying purposes.

The Tangible Archive is an example of a configurable platform–furniture made of plywood or Plexiglas modules. The furniture can be used as a surface for doing work (with work zones being reserved for particular activities), as shelves for storing materials or for projections. Cards are used for defining and loading specific configurations for the different work zones. The Mixed Objects Table is a platform for creating and manipulating mixed-media objects. As in the previous example, configuring is hardly distinguishable from proper use. Configuring and selecting textures to be painted and virtual objects to be moved on optical markers happens as part of the experimentation process.

In all the described examples, configurability includes interventions in the physical landscape of space and artefacts. The complex activity of configuring unfolds, and, therefore, should to be supported, on different levels and across different aspects of the environment: spatial arrangement (e.g. grid for fixing projection surfaces), furniture (the Tangible Archive with its modules, the table), the landscape of artefacts (which can be tagged, furnished with (hidden) sensors or (visible) barcodes), electronic components and devices (scanners, readers, connecting and plugging input and output devices), digital components and their interaction (digital infrastructure, associations of inputs, outputs and media content in the database).

This large variety of methods can provoke confusion in participants who are unable to find a rationale to deal with the new qualities of the space where they act, as well as in the designers, who miss the compositional grammar for creating their devices and arrangements. Even the weaknesses of the space offered to users (recalled briefly in the evaluation sections above) can be attributed to the lack of a conceptualisation shaping the design of tangible computing environments. We were, therefore, somehow forced to enter into a discussion of the qualities that the artefacts we were designing had and/or should have. On the one hand, this discussion has created a deeper understanding of what we are doing in Atelier; on the other, it indicates new possibilities for the design for configurability that we have not yet pursued in our research. In this section, we report briefly the first outcomes of this discussion.

6.1 Mixed-boundary objects

Most of the experiments during the first and the second trial focus on improving and enriching the presentation of the outcomes of students’ design projects to teachers, visitors and other students. The artefacts we provided for them were, in fact, used to create absorbing and dynamic environments where what they had done could be brought forth to their audience. The grid, the Tangible Archive, the Mixed Object Table, the Texture Brush, as well as the physical models and/or project plans, enriched with barcodes and/or touch sensors are all examples of boundary objects [40] and/or allow the creation of boundary objects. A boundary object is anything that can help people from different communities to build a shared understanding. Boundary objects will be interpreted differently by the different communities, and it is an acknowledgement and discussion of these differences that enables a shared understanding to be formed. While it should be immediately clear why models and plans are boundary objects, helping visitors to understand what students do in their projects, considering the artefacts we have created to support multimedia representations as boundary objects requires additional clarification. They are boundary objects, since they allow visitors to share with the students the knowledge about their design space (and the constraints and the opportunities it offers). Also, they support students in creating boundary objects to represent their work. Brian Marick [41] lists some interesting facts about boundary objects:

  • If x is a boundary object, people from different communities of practice can use it as a common point of reference for conversations. They can all agree that they are talking about x.

  • But the different people are not actually talking about the same thing. They attach different meanings to x.

  • Despite different interpretations, boundary objects serve as a means of translation.

  • Boundary objects are plastic enough to adapt to changing needs. And change they do, as communities of practice cooperate. Boundary objects are working arrangements, adjusted as needed. They are not imposed by one community, nor by appeal to outside standards.

Our artefacts support this mixture of commonality and diversity, offering the possibility to move from one representation to another, either changing the level of abstraction, or changing the supporting medium or, finally, changing the viewpoint. Several different representations that users can access make reference to one unique thing (the designed building and/or device, the planned territory and/or space, etc.).

In our approach, boundary objects are intrinsically multi-affordances objects, where commonality is supported by the emergence of one unique object and diversity by the multiplicity of affordances through which users can access and manipulate it. Considering the experiments we have done in Atelier, some of them deeply adhere to this concept (e.g. the Texture Brush) whilst others have not yet fully developed it (in some cases, any representation seems to have its own life and its links with other representations of the same object are not highlighted).

Our boundary objects, therefore, are often and should always be, mixed objects, i.e. objects coupling physical and digital qualities [39]. Mixed objects are characterised, at least, by having both physical and digital affordances (e.g. the plans enriched with barcodes), at most, by having mixed affordances (the building model painted with a digital texture). Even the concept of boundary becomes broader than in its original definition by Leigh Star: it refers to the contact line not only between different communities, but also between the physical and the digital, and, as a consequence, between the different (spatio–temporal) situations of any participant.

6.2 Openness, multiplicity and continuity

It is possible to continue this discourse at a higher level of abstraction, focussing on the qualities that mixed–boundary objects should have. One of the authors of this article, in [42], considers openness, multiplicity and continuity as indispensable qualities of what physical, digital and mixed artefacts are becoming in our everyday life. These qualities appear to be particularly appropriate for mixed-boundary objects and settings with evolving physical landscapes and activities.

Openness refers to how accessible and learnable an artefact is, and to its capability of being combined with other artefacts. Moreover, openness refers to the capability of an artefact (an affordance) to have different, potentially unlimited, ways of being used and perceived. Our experience of providing students with simple prototypes, helping to extend them and furnish them with more complex functionality is an example of openness to appropriation and use. Another crucial aspect of openness is the possibility for an artefact to be combined with other artefacts. Integrating barcodes, tags and touch sensors in physical models and diagrams helped create interactive and, in some cases, innovative combinations of physical and digital objects being perceived and used in many different ways. The Texture Brush and optical markers, applied in combination with physical objects and projections, resulted in rather intriguing kinds of mixed-boundary objects.

Multiplicity refers to the capability of a space or artefact of being made of different components having different qualities. Multiplicity can be seen in the combination of input (sensors, tag and barcode readers, scanners, etc.) and output (displays, printers, projectors, etc.) devices characterising the work-space of the Atelier students, and/or in the multiplicity of affordances offered by mixed boundary objects.

Continuity refers to the capability of moving from one affordance to another, from one representation to another, without changing artefact, without interruption in space and in time. In some sense, multiplicity and openness are contradictory as multiplicity creates distinctions and “thus boundaries between one function and another, whereas openness breaks down all borderlines to encompass all functions in one whole” [43]. One of the arguments that may guide such integrations is that the new objects populating augmented places need to combine openness and multiplicity through continuity. Continuity can be achieved by putting resources on the borders of objects so that the borders act as both separators and connectors [44].

The new types of sophisticated mixed boundary objects we are experimenting with could be achieved by putting resources on their physical borders—on the borders between their physical and virtual components. One of the challenges we are still working on is to be able to support all activities in embodied interactions without the use of conventional GUIs. Another challenge is to support participants in carrying out configurations flexibly, accountably and with ease. The integrations of physical interfaces with existing and evolving landscapes of physical objects we described, were aesthetically appealing and inspirational, but some of them were handcrafted and co-developed “on-the-fly.” Although grid and sliding frames supported the spatial configuration of projections, this still required artistry and bricolage (which the architecture students are quite capable of). The furniture we designed is with wheels and modular, but more flexible ways need to be found of integrating it with electronic components. Integrating sensors into artefacts has improved since our first trials, from wired prototypes to wireless components, but still need to be customised into (sometimes) fragile solutions. End-user configuration at all the levels we have identified poses a series of challenges for architectural, industrial and interaction design (including the need for robust and open electronics components).

6.3 Configuring as staging mixed places

The potential of physical interfaces reaches beyond “mere embodiment.” They provide people with the means for producing configurations that change spatiality and interactivity, and a physical landscape in ways that help users experience, explore, present and perform. At the beginning of the field trials, the space was non-intentional and had to be appropriated. This was also part of the pedagogy, assuming that a perfectly furnished space is often not the best solution for creative work. Students need to appropriate the space, struggle with its constraints and find their own interpretation and setup. This is why they found the space almost completely empty, apart from an infrastructure of networks, furniture, grids for projections, tag/barcode readers, computers and other electronic equipment. They were asked to bring their own artefacts—pictures, video material, scale models, diagrams and collages. With these resources at hand, students configured and reconfigured space and artefacts to accommodate diverse activities—from browsing through pictures, to discussing a design concept or performing a scenario of use. We can understand these configurations as forming an evolving set of temporary and, in some ways, ephemeral layers onto this neutral, almost empty environment.

This captures the aim of current research on enriching the physical space, which is not solely to provide new functionalities for participants, but to transform their social experience at its roots. The space metaphor does more than provide a resource for analysing human behaviour and designing for it; it also shapes the language through which we speak about ourselves. Just as it allows designers to describe relationships in social interaction, it provides a common framework for those actually engaged in interaction. Harrison and Dourish [45] argue that a place is a portion of space “invested [through language practice; authors’ remark] with understandings of behavioural appropriateness, cultural expectations and so forth.” When people share an enriched portion of space and a language to talk about their experience, they transform the former into a place. So what could mixed places be? They provide qualities, some of which cannot be found in physical space and are utterly new. One quality of mixed places that emerges from our trials is the capability of being reconfigured dynamically and radically. The configurability of a space depends on its layout, the design of the infrastructure and the design of the artefacts that populate it. With regard to the configurability of artefacts, we have argued that they would have to combine openness and multiplicity through continuity, producing the right interplay between infrastructures, mixed objects and activity. As to the configurability of a space, we could learn from good architectural design that often plays with an ambiguity in the relationship between spatial configuration and functional program, where “the allocation of functions or uses is malleable, they are fitted into the spatial configuration. While some of them find ample space, others might have to be squeezed in, overlap, extend into neighbouring spaces, thereby creating “natural” connections or meeting “fixed” boundaries. This not only allows to suspend or transgress the usual hierarchy of functions and rooms. Also, the boundaries between interior and exterior space are designed as permeable and fluent” [13].