Keywords

1 Introduction

This research converges, partially, with the current phase of a doctoral thesis entitled Augmented narrative: contributions of digital technologies for information design, where exploring augmented narratives for classic examples of information design is the main task, aiming to confirm the relevance and feasibility of using augmented reality in this design field.

In this paper we present some of our experiments intended to test and confirm the interest and viability of incorporating augmented reality (AR) technologies in cultural mediation for information. We aim to ascertain if there is an added value in the application of augmented reality in information design artefacts from a semantic point of view, and explore in an open way hypothesis of appropriation of this technology in which the combination between real and virtual configures new semantic relations.

Observing the strategy and phases for narrative representation in the information design approach - Circle of datascape navigation [14, 18] it is possible to circumscribe, within this specific study, the testing of augmented narratives to the “results presentation phase” of that approach. By exploring augmented reality in the communicational phase of narrative representation, it is also intended to provide ways of personal appropriation of information, that according to Wurman [20] are essential to transform information into knowledge.

Methodologically, this study belongs to the paradigm of Research through art and design [8], specifically in development work which aims at adapting or appropriating a technology for a new use. New knowledge is created through an action-reflection approach, in which emphasis is on the research objective, not the project solution [7].

In this paper we present the experiments done based on the reinterpretation of the program embodied in the Figurative Map (1869) by Charles Minard, in order to recognise and openly systematise different ways of adequate augmented reality to information design. Figurative Map was selected considering that it constitutes a traditional information design object, widely recognized as exemplar in the culture of the discipline [10, 16]. Rooted on the experience observations, different augmented reality functions regarding the physical object are acknowledged, which can potentially expand traditional forms of information design. This proposal was built in a parallelism with the study of the relations of text and image, in this case coming from illustration, namely from Sophie Van der Linden study [12].

The conclusions of these research shall guide its future application in the project designobs.pt - Towards a Portuguese Design Observatory: Models, Instruments, Representation and StrategiesFootnote 1, namely in the construction of information artefacts about aspects of portuguese design in which augmented reality will be an important narrative and semantic component. Therefore, we do not intend to test AR to inform directly over reality but over an interpretation of reality - in this case, numerical data about the portuguese design ecosystem. With the experiments carried out using Minard Map, we aim to discuss the concepts underlying the application of AR to information design, confirming their interest in a phase that precedes the application of this technology to the DesignObs. project.

2 Augmented Reality

Bolter & Grusin describe augmented reality (AR) in opposition to virtual reality (VR). Fully independent of the real world, virtual reality is so opaque that it becomes transparent. There is a simulation in which the media disappears designated as immediacy, which alludes to the notion that “the medium could erase itself and leave the viewer in the presence of the object represented, so that he could know the objects directly” [3].

“(…) virtual reality is complete in itself. It is, in fact, Baudrillard’s simulacrum, bearing no necessary resemblance to anything that exists in the physical world. Ironically, the apparent immediacy of such a virtual reality comes from its total opacity. (…) The purpose of this opacity to the physical world is to provide the user with a perfectly transparent world generated by the computer” [3].

Augmented reality doesn’t substitute the real for an immersive environment. Rather than aspire to transparency, augmented reality asserts itself as media: “In laying icons, texts, and images over visible objects in the world, augmented reality frankly admits that it is a digital medium interposing itself between the viewer and an apparently simple and unitary physical world” [3].

By commenting or annotating on the real, augmented reality adds layers of meaning and interacts over it. “Culture augments reality. In fact, human beings inhabit a world augmented by this symbolic layer which created and continues to create. (…) Any cultural production is Augmented reality, and in this sense, design is a net contributor, even responsible for the cultural mediation which promotes techno-science for day-to-day usable things, likely to enter into the semantic game regarding the unfolding of the human being” [4].

AR technologies can be divided in three main categories, distinguished from one another by the display technology used to overlay augmented reality onto the physical world: Hand-Held Displays (HHD), Head-Mounted Displays (HMD) and Spatial Augmented Reality (SAR). [11, 15, 17]. Due to the democratization of the smartphones and tablets, Hand-Held Devices are the most common instruments for implementing AR, having originated Mobile Augmented Reality (MAR) [5]. There are two common strategies regarding AR in Hand-Held Devices: (1) through the use of visual markers on the objects that are to be augmented; or (2) through the location-based tracking, mostly used in outdoor settings. Head-Mounted Displays are the most developed and available technological solution for wearable AR and its technological approaches are: projection, retinal, optical and video. Spatial Augmented Reality (SAR) lies on the overlay of digital images over a physical area. This overlaying can be achieved either through video, projection or holographic/optical displays.

Technologically, the alignment of augmented reality goes back to the Reality-Virtuality Continuum by Milgram and Kishino [13], a taxonomy that, for the first time, integrated the real and the virtual environments in a same continuum, classifying augmented reality and augmented virtuality as mixed realities. In 2001, Azuma et al. characterized an augmented reality system according to three properties: 1) Blends real and virtual, in a real environment; 2) Real-time interactive; 3) Registered in 3D.

The proper alignment of virtual objects to the real time environment, known as registration and tracking [2] are the sensible issues in AR applications [15] and, without that correct alignment, the illusion that the virtual object participates in the real space gets severely undermined. Augmented reality SDK (Software Development Kit) came to facilitate several integrating aspects of applications that use this technology, namely registration, tracking and rendering of AR content. In a comparative study, Amin and Govilkar [1] describe several development kits available to create AR applications, including Vuforia, that uses image recognition techniques to detect predefined image targets which can be images or 3D objects. These targets are loaded to a management system web-accessible which assess them according to their ease of detection, without needing special markers.

Vuforia was used in our Mobile Augmented Reality (MAR) experiments because of its technical advantages that include the fast detection of targets and a very good capacity of tracking, even in low light conditions, distance or in which the target is partially covered. Another important reason is related to its easy integration with Unity - a game engine very efficient in the construction of multiplatform interactive applications.

3 Method

The selection of Figurative Map (1869) by Charles Minard to test augmented narratives was due to its acknowledgment as excellent in information design culture. “Charles Joseph Minard’s data-map describes the successive losses in men of the French army during the French invasion of Russia in 1812. Vivid historical content and brilliant design combine to make this one of the best statistical graphics ever” [16].

Figurative Map (Fig. 1) shows through a flow chart, currently designated ‘Sankey diagram’ the catastrophic life losses suffered during the Napoleon’s army campaign from Kaunas to Moscow. According to Costa [6] the designation ‘Sankey diagram’ it’s resultant of a diagram about energetic efficiency created in 1898 by Mathew Sankey, an irish engineer. Minard drew its ‘Sankey’ almost thirty years before. However, Minard didn’t invent the use of flow lines in a map, in accordance with Friendly [9]: “That honor belongs to Henry Drury Harness (1837), showing passenger traffic and the flow of goods in Ireland”.

Fig. 1.
figure 1

Carte Figurative des pertes successives en hommes de l’Armee Français dans la champagne de Russe 1812-1813. Map drawn up by Charles Minard in 1869.

Full size image

The decreasing size of the army, that initially counted with 422.000 men, it’s highlighted by the narrowing brown line that represents the departure to Moscow and by the black line that marks the return to Kaunas, finishing with 10.000 men. Minard depicts in a single image multiple dimensions: geographic localization, number of soldiers in each map position, temperatures endured by the army and time passage during the retreat.

Beyond the reference to low temperatures during the return, Minard doesn’t determine death causes over the several points of the map where the flow line drastically narrows, leaving that explanation to the reader’s curiosity, in what Willers [19] designates as “open ending”. Based upon this premise, three reinterpretations of Minard’s map were conceived, using augmented reality as a means to integrate the original map with the reinterpretations of its program.

4 Results

4.1 Experience 1 – Tridimensional Terrain

In the first experience a tridimensional modelling of the geographic terrain depicted in Minard’s map was tested overlapping the original map, through mobile augmented reality. The tridimensional modelling was done using Cinema 4D, combined with 3D Map Generator and Google Maps for the creation of the heights map. The augmented reality component – experience exported in the form of an android application – was done using Unity (Vuforia), allowing to overlap the 3D model to Minard’s map set as image target (Fig. 2).

Fig. 2.
figure 2

“FigurativeMap.apk” app screen capture (experience 1).

Full size image

The modeled area is too extensive, it currently covers three countries: Lithuania, Belarus and Russia. The map texture is generated through a screen capture of the same geographic area in Google Maps, contributing to a very homogeneous terrain with no details, such as small lakes and rivers. The experience allowed us to observe that the total occultation of the real object by the virtual has annulated the augmented reality purpose, emphasizing the need for searching means to completely integrate reality and virtuality.

4.2 Experience 2 – Additional Information

The second experience consisted in marking battles and relevant events in several points of the map, augmenting it through the creation of new layers of information that wasn’t on the original map, but could provide leads to its comprehension. Data collection and processing about the French Army Campaign to Russia was limited to the content used in this experience, which was retrieved from the dynamic visualization When Napoleon Ventured East, published online by Russian news agency Tass. Additional information, composed by explicative texts and graphs comparing the number of men and guns of each army in successive military confrontations, started by being introduced in tridimensional panels located on the respective map points and activated through independent image targets.

Initially this test came to expose the technical difficulties in recognizing small targets. On a further development (Fig. 3), in which every informative panels were associated to Minard’s map, despite the well succeeded registration, the number of close panels produced entropy on visualization and generated shadow projection on the panels.

Fig. 3.
figure 3

“Battle-Map-3.apk” app screen capture (experience 2 – virtual buttons).

Full size image

In order to dissolve the entropy caused by the simultaneous placement of information, virtual buttons were created to activate each panel. Despite allowing a better integration between real and virtual, the use of virtual buttons in mobile devices presents some disadvantages, namely the need for finding the best angle to detect the virtual button and, simultaneously, be able to visualize the associated panel (Fig. 3).

Fig. 4.
figure 4

“Battle-Map-4.apk” app screen capture (experience 2).

Full size image

The following step consisted in replacing virtual buttons for buttons which interaction is done through the device screen and visible with augmented reality. A transparent material was added to the panels, making the original map completely visible in loss of additional information (Fig. 4).

The possibility of transforming any object into a button through its programming opens several creative possibilities for information design. In this case only information panels were used, but the same buttons could activate diverse materials: tridimensional graphics; sound; animation; tridimensional models; terrain maps; etc. Buttons allow to provide a large amount of information associated with the map without being immediately visible, but interactively discoverable.

4.3 Experience 3 – Further Reading

The Further Reading experience consisted on associating several objects regarding Napoleon’s Russian Campaign to the original map, locating that materials on the corresponding geographic points (Fig. 5). These materials include painting, photography, poster, movie, audio, songs, personal reports, as is the case of De Ségur, and articles excerpts. With this experience, an interactive multimedia piece was tested, approaching the cultural experience of war.

Fig. 5.
figure 5

“Further-Reading.apk” app screen capture (experience 3).

Full size image

Technically, the third experience presents issues that can be corrected or optimized. Texts reading implies moving the mobile device away from the image target (Minard’s map) which causes the text to be deactivated. The visualization of each object in full screen through selection and enlarge option could facilitate the app interaction and the experience of each material. With the script AudioSource.PlayClipAtPoint it is possible to play an audio clip at a given position in world space, contributing to a more profound fusion between real and virtual, as well as the shadow projection of virtual objects.

Conceptually, it is necessary to question if a piece like this can suggest a softening of the war horror. However, in this research phase it is considered essential to assess all technology appropriation scenarios, reviewable upon curatorship.

5 Semantic Relations Between Real and Virtual

When the virtual component is not redundant regarding the real environment to which it is associated, it can incorporate several functions. In any case mobile augmented reality is always a means to open, unveil, display something that is hidden in a virtual dimension, visible only through a suitable device. In that sense, Bolter and Grusin [3] are correct when they refer that augmented reality is a remediation of the windowed style of the desktop interface. In opposition to the sequential logic of the book, there is a depth logic or mise-en-abyme logic. Although it is possible to virtually add several materials to the physical object, as in Further Reading, contributing to an information ‘flatness’, the way to trigger the virtual component is always the real. In that perspective, the main instance is always the real and the secondary instance is the virtual, since the second depends invariably on the first.

Based on the previous experience observations, different augmented reality functions regarding the physical object are acknowledged in an open way, which can potentially expand traditional forms of information design. Although they are not entirely extrapolatable, these functions borrow some illustration concepts, specifically image and text functions defined by Sophie Van der Linden [12] who determine repetition (fonction de répétition), selection (fonction de sélection), revelation (fonction de révélation), to complete (fonction complétive), counterpoint (fonction de contrepoint) and amplification (fonction d’amplification).

The main difference between the logic of text image relations studied in illustration and the logic of the relations between real and virtual involved in augmented reality is that, while text image relations where formulated considering the book medium which builds on the passing of time simulation afforded by the sequence of pages and the reading mode, in the case of augmented reality instead of sequence, the medium promotes virtual deepness built with digital information layers which superimposes reality. Deepness in this context refers to the mise-en-abyme conformation enabled by the medium, and not to the reading depth. Another difference between the construct of image text relations in illustration and the relations between real and virtual we now propose is that while the first case concerns the relation of two different modes (image and text) embodied in the same medium, the second case concerns the relation between several modes (image, text, video, sound, discourse, etc.) and potentially different media. In that sense, the formulation of a dichotomy between real and virtual is also operational in order to understand how augmented reality shapes content.

Based on this premise, virtuality can potentially perform the following functions regarding reality:

5.1 To Complete

When virtual intervenes over the real, providing additional information and fulfilling gaps or omissions. Like the function of completing in illustration, in the semantic game between real and virtual these two instances can be integrated to be understood as a whole.

5.2 To Embody

When virtual is used to ‘make more real’, to simulate the object’s tangibility, getting closer to the simulacrum logic characteristic of the virtual reality.

5.3 To Explain

When the virtual is intended to explain or deconstruct the information conveyed in the real instance, making it more comprehensible.

5.4 To Duplicate

When the virtual adds other ways of perceiving the same information:

Synesthesia.

Creation of a bond between information that is presented visually with other senses, so that information can be assimilated through different channels.

Quantitative/Qualitative Approach.

Transformation of a quantitative visualization in a qualitative approach, or vice-versa.

5.5 To Communicate a Visual Metaphor

Illustration of the data presented in the physical artefact through a visual metaphor.

5.6 To Counterpoint

When the virtual component is used to communicate a contradictory perspective to the one being presented by the physical artefact, in order to convey more than one vision of the same information.

6 Conclusions

Results of the experiments carried out, although at an early phase, enabled us to extract useful clues about the application of augmented reality technologies to information design. Based on the first experience, the need to explore ways of better integrating reality and virtuality got visible. Integral occlusion of the real object by the virtual one annuls the augmented reality purpose and distorts it – the experience is no longer augmented reality, but it’s neither considered virtual reality. The attribution of audio fonts to specific locations, as well as shadow projection of virtual tridimensional elements are means of artificially simulating a more profound fusion between real and digital. In the virtual buttons experience, interaction with the mobile device and, simultaneously, with the physical object revealed itself laborious and unnatural, distracting the user from content. In this regard, it is essential to test other augmented reality categories in which the display positioning is more transparent to the user, such as head-worn and spatial displays. Buttons allowed to make available or hide information associated with the physical object to be discovered. This game of invisibility and discovery can provide an adapted reading to each reader, depending on the reading level he intends to do.

The enunciation of possible augmented reality functions in its relationship with the real was inferred from the described experiences and does not intend to cover all possibilities. It is acknowledged that this is an open model to be reconsidered and reformulated through several action-reflection iterations and fostered through the narrative study.

Future work might be framed according to three main guidelines: the semantic relationship between real and virtual, augmented reality technology and user experience. There is a prior need for further reflection about the semantic relationships between real and virtual under the narrative perspective, which demands the practical exploration and test of each one of the proposed functions in the last section. It is also important to test other display techniques, namely the Head-Mounted Displays (HMD) and the Spatial Augmented Reality (SAR) categories, being fundamental to understand each category’s impact in narrative appropriation and overall user experience.