Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The AR Lab started as the AR  +  RFID Lab of the Royal Academy of Art in The Hague, The Netherlands, in which applications of augmented reality and radio frequency identification (RFID) in art and design, society and commerce have been researched since 2006.

In the past years we have seen these innovative technologies being used as a creative medium for autonomous artists, as animated prototyping tools for furniture makers and interior and urban development architects, as a virtual tool for the digital visualization of valuable cultural heritage and as geo-data visualisation tool. The applications of augmented reality are as diverse and unlimited as ones imagination [1].

It is our experience that every field of study has its own way of using new technology in its research. The creative character of Augmented Reality—creating something without matter—makes it exceedingly suited for art and design at an art academy. At the Royal Academy we have the special opportunity to experiment with high-quality technology from the Delft University of Technology, a partner in our AR Lab.

The Lab develops projects in collaboration with students and artists, on invitation of museums and/or for educational purposes. The work is presented at the involved museums, at innovation events for companies, at scientific conferences, in educational institutes and during art- or design manifestations. Furthermore, the lab regularly receives visits from students from other academies and universities, artists, architects, designers and interested companies that want to get started with the new possibilities augmented reality creates.

In this chapter we discuss three examples in the domain of cultural heritage in which we tried to enhance the interaction with work of arts through the use of innovative visualisation techniques like augmented reality. Our experiments reported here- and many more—has led to a list of lessons learned that we present at the end of this chapter. The idea behind these lessons is that we and other AR artists, designers and researchers can use it as initial material to design better AR installations. A next step would be to use the UX evaluation metrics proposed in Chap. 9 to evaluate our future AR installations.

2 Pre-industrial Earthenware from 1250 to 1550; Museum Boijmans van Beuningen, Rotterdam, The Netherlands

The project “Sgraffito in 3D” has made the late medieval pottery collection of Museum Boijmans Van Beuningen accessible to a larger audience through 3D reconstruction.

Sgraffito is the term to describe earthenware pottery, in which with the aid of a sharp tool, decorations have been scratched into a thin layer of clay slip. Since the Middle Ages, this centuries-old oriental decoration technique was introduced into Western Europe of Persia and the Byzantine Empire. In the fifteenth and sixteenth centuries, potters in the Netherlands applied this technique onto simple domestic earthenware. The Van Beuningen-de Vriese collection, which is part of the Boijmans Van Beuningen Museum since 1983, contains a collection of late medieval sgraffito earthenware. These examples have been produced between 1450 and 1550 in several potteries in the Netherlands. The museum also owns a small collection of Iranian Sgraffito earthenware.

Our goal was to increase the attention for this collection [2]. Not many people are interested in old pottery. Most visitors of the museum don’t feel related to old ceramics and they are not motivated by the old dishes themselves to have a closer look at its peculiarities. When the curator Historic Design Drs. Alexandra Gaba-Van Dongen asked artist Joachim Rotteveel to enhance the experience for visitors, he designed in co-operation with the AR Lab a way to make interaction possible: visitors could touch, take-up and manipulate 3D-printed replicas. A 3D-archive let visitors play with digital representations of our cultural heritage, which were also available for research and downloading online. The augmented reality installations offered visitors tactile interaction with the replicas and direct visual and aural access to background knowledge of the valuable cultural objects. Also, the website www.sgraffito-in-3d.com was developed where one could see all ceramics (not only the ones used in this experiment) and gain information. This functions as a virtual museum [3].

Seven dishes, the “protagonists” were CT-scanned (Fig. 11.1) and from the CT scans virtual copies of the dishes were made. The virtual copies were 3D printed and thus became physical copies. They were exposed very close to the original medieval ones that were stored behind safety glass. The 3D printed dishes were chained and the audience could touch and manipulate them. The augmented reality part came in with our AR installations explained below.

Fig. 11.1
figure 1

CT-scanning and resulting CT scan. Mark the fractures repaired in earlier days

Full size image

2.1 The 3D Pop-Up Book

The 3D Pop-Up book is an AR book in which text and music is attached to the iconography of the Sgraffito decoration of the dishes. The books (English and Dutch versions) were situated in front of an LCD screen with a camera on top and next to each book were headphones available. The camera image looking to the book was displayed on the LCD screen with on top of that the (augmented) 3D graphics (Fig. 11.2).

Fig. 11.2
figure 2

A 3D pop-up book showed—in AR—the dish and its ascending decoration

Full size image

For each of the seven protagonists, the book offered two pages next to each other. On the right page one could see the decoration in a 2D print, on the left page text related to the iconography of the decoration could be read. An AR marker was printed on the left page. When the camera recognized the marker three things happened:

  • Music was played from the era the dish was made and related to its decoration

  • The dish appeared on the LCD screen in 3D on the right page of the book

  • The Sgraffito decoration was released from the dish, drifted above the dish and turned around and returned again to the dish. This animation looped

2.2 Integration of the Virtual and the 3D Printed Shards

The original lines along the dishes were once broken and restored in their time (the ceramics stem from 1250 to 1550) were clearly visible in the CT-scan. From each plate we 3D printed a shard. When holding a 3D-printed shard in front of a webcam it reveals its missing pieces on the monitor and thus forms a complete virtual dish. The 3D fragment acts as interface for the virtual object; when you move the shard in your hand you move both the physical and virtual objects simultaneously as if it were one integrated object.

The shards were chained and could be hold in front of the camera and so create a complete dish in AR on the screen (Fig. 11.3). Even children understood the concept (Fig. 11.4).

Fig. 11.3
figure 3

3D printed shards create a complete virtual dish in AR on a monitor

Full size image
Fig. 11.4
figure 4

Young children understood the interaction between shard and monitor immediately

Full size image

2.3 Spatial AR “out of the screen”

At the opening of the exhibition we set a table with a cloth embroidered with black and white markers. A visitor wearing our AR headset could see the medieval earthenware on the table (Fig. 11.5). With cardboard AR markers one could add dishes and change their order [4]. We used a beamer to project the scene on the wall behind the person wearing the AR headset to show the other visitors the augmented scene.

Fig. 11.5
figure 5

Laying the table with medieval pottery in spatial, immersive augmented reality

Full size image

2.4 Discussion

Our goal was to enhance the connection with “old pottery” by giving the visitors an interactive experience. We found out that:

  1. 1.

    Screens (monitors) with moving images generate attention; visitors tend to have a look on a screen on which action is to be seen. We saw that visitors unintentionally were attracted to our AR 3D pop-up books because the marker on a page of the open book generated the moving virtual dishes [4].

  2. 2.

    A book is a very well known object: it is not necessary to explain that one can turn a page. That is why visitors seemed to be “seduced” to interact with the virtual images on the screen, by turning the pages and by doing this provoked new images even without thinking. The motivation to act came from inside; intrinsic curiosity led the visitors to stand still, turn pages and after seeing the plates appear, they were persuaded to put on the available headphone.

  3. 3.

    The delicate music heard through the headphones connected with the monitor on which the pop-up books could be seen, was for most visitors very particular. The music originating from 1250 to 1550. This single-person experience enhanced the concentration with which the visitors looked at the ascending and descending decoration of the virtual dish, meanwhile reading the text that was providing a complementary and sometimes terrifyingFootnote 1 information. Thus a rather high level of immersiveness was generated: this was for most visitors an eye-opener; one could really get “involved” with a piece of old pottery [5]!

  4. 4.

    Since the 3D printed shards were placed in front of a monitor and each “hard copy” shard generated a sequence of centrifugal and centripetal virtual shards on the monitor (thus completing the dish and subsequently falling pieces), the relation between the “hard copy” fragment and the virtual fragments was immediately clear for each visitor. This was especially true for very young visitors.

  5. 5.

    From earlier experiments (Milano, see below) we learned that when using AR headsets in a crowd, the right way to attract attention is to peek the augmented scene from the headset and project this on a wall or on a very big screen [5].

3 Showing Virtual Furniture in Salone del Mobile, Milan, Italy

Every year in April in Milano, Italy, a huge international furniture and interior design fair is held for 10 days, the so-called Salone del Mobile. Design academies and designers from all over the world try to attract the attention of the international press and “hot-shots” in this field. Italy and especially Milano is world famous for its innovative design, so for everybody who has an exhibition during this week there is a lot at stake.

The AR Lab wanted to research if it is possible to show virtual furniture to a large audience with spatial AR, and furthermore, how the audience receives this. Will the AR installation enhance the connection between the professional audience and the designs of our students [6]?

We rented a house in Milano and prepared the big living room for our virtual furniture show. We placed a grid of markers meticulously precise on the walls and the floor. The cameras attached to the AR headsets could identify the markers and their position in the room, making it possible to put virtual furniture at any desired place in the room.

3.1 Augmented Furnishings (Cupboards, Tables and Chairs)

We had two headsets with two backpacks available (Fig. 11.6a). They were rather heavy and not really comfortable. Figure 11.6b shows that the backpack was for a child too heavy to carry; a team member supported him. The weight was due to the heavy duty laptop, AR glasses control box and batteries.

Fig. 11.6
figure 6

(a) The two headsets with backpacks. (b) Sometimes a carrier was welcome

Full size image

The AR furniture could be seen from all sides, this caused some visitors to go on all four to look behind and beneath the virtual furniture (Fig. 11.7).

Fig. 11.7
figure 7

Visitors engaged with the virtual furniture in Salone del Mobile

Full size image

3.2 Animated Textiles

In the centre of the room we attached over 20 printed textiles the size of a towel at a pillar. Attached to these textiles were RFID tags. When a visitor equipped with our headset, put on a glove with an RFID-reader attached to the glove, he could see in augmented reality, a large banner of the designed textile from the ceiling to the ground, about 3.5 m high and 4 m wide. These augmented prints changed with the real prints he/she touched with the glove.

3.3 Discussion

First of all we have to keep in mind that this exhibition was held in a nervous Italian environment, a huge fair spread all over Milan, in which every exhibitor was in competition with the other because there was too much to see. Therefore the visitors generally want to be able to make an instant decision when entering a venue: “this is worthwhile, so I’ll spend some time”, or “I skip this and hurry to another venue”. Having to put on the heavy backpack, to adjust the headset and to give some instruction takes at least a few minutes. We had to find a way to track attention of potential visitors. Screens at the entrance showed the living room with the visitors queue and the augmented scenes from the headsets. The students at the entrance door wore fake headsets to draw the attention to AR. Hence we drew a lot of attention (Fig. 11.8a).

Fig. 11.8
figure 8

(a) Visitors waiting to wear the headset. (b) Duplicating chairs is easy

Full size image

In this experiment we encountered the following problems:

  1. 1.

    An augmented reality headset is single-person equipment. This is quite a contradiction for a public space in which we want to involve as much persons as possible.

  2. 2.

    Visitors do not like to wait too long for any experience; this is equally true for a quite “exotic” and immersive experience with sophisticated equipment.

  3. 3.

    There were difficulties with the alignment, i.e. the placing of the virtual objects on the desired place.

  4. 4.

    As in normal exhibitions, one needs a professional curator to do the placements of the objects to make a coherent view see (Fig. 11.8b).

  5. 5.

    Virtual objects or scenes might jitter, jumping a little up and down or appear and disappear, based on unpredictable human motions and camera occlusions.

  6. 6.

    People like to be amazed but demand an instant explanation directly following this experience. Sometimes this necessity has to become technical, e.g. explaining that when a marker is blocked virtual furniture disappears or flies away.

4 Commenting Art in the Sculpture Garden of the Kröller–Müller Museum, Otterlo, The Netherlands

The Kröller–Müller Museum named after Helene Kröller–Müller (1869–1939), is settled in the woods. The sculpture garden was opened in 1961 with works by Rodin, Moore and Hepworth, Snelson, Christo, Serra, Jean Dubuffet, and others. The works of art are spread across the entire terrain, the one clearly visible, the other secluded in a remote corner or hidden among the rhododendrons. Each piece has become inextricably linked with its place in the sculpture garden.

For our AR installations we were allotted a part of the Sculpture Garden. The Lab presented a large augmented reality installation for the “Sweet Summer Night Illusion” exhibition in 2009.

The assignment of the museum was to comment on some of the twentieth century sculptures with tools of the twenty-first century in line with the theme of the exhibition [7]. Since the theme of the exhibition was Illusion, we could run pretty wild. Part of our AR images were related to Hieronymus Bosch’ famous triptych the Garden of Earthly Delights, with angels and monsters all around the scene.

To work in the open air with augmented reality is quite a challenge, let alone the largeness of even “our” relatively small area.

For this event, the Lab created unique 3D content and developed innovative augmented reality equipment, such as walkers (for elderly people) with screen-based AR systems mounted on them, and a large rotating screen showing a 360° panorama of the sculpture garden. Mounting laptops on walkers seemed us to be a humorous interface. For rainy weather we prepared an AR headset experience inside the museum.

4.1 Walkers and the Garden of Earthly Delights

With Mac books mounted on walkers, people could make a tour in “our” part of the Sculpture Garden. Some markers were placed on twigs of the trees with clothespins and other markers were attached to small sticks in the ground. The camera caught the markers and on the screens fantasy animals were seen by which we were augmenting the bushes (Fig. 11.9).

Fig. 11.9
figure 9

The AR walkers in the Kröller–Müller Sculpture Garden

Full size image

4.2 Big AR Screen Commenting on the Sculptures

From our spot in the garden we could see two sculptures, one from Mario Merz, Igloo di Pietra and one from E. Dodeigne, Couple. When preparing the AR experience, we figured that people would wonder what was inside this Igloo. When they turned the big AR screen towards the Igloo, they could see the original sculpture and on the screen what could have been inside in augmented reality (Fig. 11.10).

Fig. 11.10
figure 10

Big markers can be caught by the AR camera over long distances (a). Igloo di Pietro, (b). visitors, (c). Couple

Full size image

Researching in Dodeignes oeuvre we found out that Dodeigne wrote in his diary he wanted to have more stone couples at that spot. We made an extra virtual couple and when turning the screen towards the stone couple, one could see two couples on the screen: one real and one added in augmented reality. The camera mounted on the large screen captured the big markers next to the real sculptures, in order to overlay the augmented content.

4.3 Spatial AR Inside the Museum

To be prepared for rainy weather we placed one installation inside the museum in an indoor garden: looking with our headset through the windows towards a cube of markers outside, visitors could see a sculpture of a head, which consisted of blocks moving forward and backwards (Fig. 11.11). The computer of the headset was also mounted on a walker, so one could see this virtual head from different angles: aside, in front and on the back (Fig. 11.12).

Fig. 11.11
figure 11

A huge virtual blockhead with animated blocks could be seen through AR glasses

Full size image
Fig. 11.12
figure 12

Inside the museum an AR headset was used with the backpack on a walker

Full size image

4.4 Discussion

Visitors were immediately attracted to the walkers. What we hoped came out. The anachronistic interface; high tech equipment mounted on a walker for elderly people in a Sculpture Garden with sculptures from the nineteenth and twentieth century. It generated joy; young and old wanted to walk with the walkers. The images people saw on the screens were a surprise for most visitors [8]. The big screen was very easy to use: people intuitively turned it around and saw the augmented sculptures on the screen.

Inside the museum—although the weather was good—the “blockhead” was impressive.

In this experiment we encountered the following issues:

  1. 1.

    Visitors were amused to walk with the walkers and found their way along the markers in the trees and on the lawns. However; there was little “click” with the theme of the exhibition, unfortunately. The excitement of the new technology overshadowed the link with the theme Illusion.

  2. 2.

    Our art students related the augmented reality fantasy creatures in the bushes to the Garden of Earthly Delights theme, but the visitors did not notice the association. More narrative information was needed.

  3. 3.

    Commenting the sculptures got the highest appreciation: this really added information and created a lively discussion among the visitors.

5 Lessons Learned

From the AR installations described above we learned.

5.1 Boijmans van Beuningen

  1. 1.

    Screen based AR is a low cost replacement of HMD based AR and can be fruitfully used to introduce the topic at hand and the AR technology itself.

  2. 2.

    People tend to like manipulating virtual objects or at least have some influence on them.

  3. 3.

    Augmented reality can be fruitfully used to attract a broad public to displays of cultural heritage. Its inherent narrative power is huge.

5.2 Salone del Mobile

  1. 1.

    The augmented view can be peeked from the tracker camera and displayed on a beamer or screen to let the other visitors see through the user’s eye at the same time.

  2. 2.

    Positioning virtual objects in the air covers up for misalignment. We met some troubles in positioning the virtual objects but we found a way to overcome this by animating the object. The visitor did not see just a virtual chair, but saw in AR an animation in which all parts of a chair (legs, backseat, elbow rests) were falling from the sky and form a complete chair on the floor. This led us to:

  3. 3.

    Motion of the virtual objects covers up for misalignment and jitter.

  4. 4.

    Manipulation of real objects can influence—through RIFD—the virtual world. This is “magic” for many people.

  5. 5.

    Design discussions are more vividly using HMD based AR as each user can now individually select his (“the best”) viewpoint.

  6. 6.

    Headset based AR is at its best when a full immersive experience is required and people can walk around objects like chairs and tables, but also within larger objects; buildings, molecules, DNA.

  7. 7.

    The “empty” third dimension in the air around us, is very useful for information display and interaction and detaches the application from gravity, we can have art in the air. There is a lot of space up there.

5.3 Kröller–Müller

  1. 1.

    For outdoor AR it is necessary that the ambient light intensity and the intensity of the LCD displays on the HMD is in balance (automatic sunglasses).

  2. 2.

    The relationship between the real world objects and the added ones should be strong and convincing; otherwise the AR work becomes just a gadget. One should not forget the story telling or design of the augmented scene. This is more complex then in a normal situation.

5.4 All Experiments (Including the Ones Not Described Here)

  1. 1.

    The collaboration between researchers in the area of image processing with artists, designers and curators appeared to be very fruitful and has lead to many amazing productions and exhibitions.

  2. 2.

    Not only humans interact with humans in and via “cyberspace” also real, physical objects communicate with each other, with or without knowledge of people. People tend to understand this quickly.

  3. 3.

    Design packages such as Cinema 4D make design with animated figures possible. Most design packages like Cinema 4D don’t allow large plots. For real 3D animated films with large plots, game engines (like Unity) must be used.

  4. 4.

    When adding sound to virtual objects, this adds to their attention drawing and pose tracking.

  5. 5.

    More image processing on the tracker camera is useful, e.g. to segment the users hand and fingers to make unhandy data gloves superfluous; keyboard interfaces are useless.

  6. 6.

    Segmenting—with image processing techniques—moving objects such as people enables virtual objects to encircle them.

  7. 7.

    Standard heavy duty gaming laptops are heavy to wear but enable easy connections to new interaction devices such as the Wii. We wait for diminished equipment.

  8. 8.

    By applying VR design techniques, i.e. also modelling walls, floor and ceiling, virtual objects appear real and real objects appear virtual.

  9. 9.

    Life video streams inside the virtual world give a tele-presence awareness.

  10. 10.

    Augmented reality books of all experiments can be used as a way of archiving the temporarily AR installations. One can show them again at any time.

6 Conclusions

Augmented reality in the artistic and cultural domain might be used as a tool for artists (fine-art and design students) and as a medium to enhance the relationship with the collection of a museum.

For all applications the cooperation with students from the Delft University of Technology was fruitful. Partly in separately financed projects, partly financed by projects run by the royal academy they developed the technology that was used for our systems [9–18]. Thanks to this we were able to use cutting edge—albeit not yet consumer friendly—equipment [19, 20]. However, what turned out to be most valuable was the sincere artistic approach with which the students (with help from our Lab) were engaged to give visitors an interactive experience, which causes a special relationship with works of art [21, 22]. To our opinion researching, rethinking, refining and redefine interaction with innovative visualisation techniques will continue. AR has just begun.