Keywords

1 Introduction

As computing technologies advance, Interactive Digital Storytelling (IDS) researchers anticipate how the field will evolve [57]. With the rise of the Metaverse, a fully immersive virtual world that blurs the lines between the digital and physical realms, exciting avenues open up [3]. The Metaverse uses various technologies across Extended Reality (XR), including Virtual (VR) and Augmented Reality (AR), increasingly combined with Artificial Intelligence (AI), among others, to build virtual spaces and simulate real-world factors. Further, it integrates the physical components in the digital world for users to experience, create, and interact with people and places.

As technologies continue to shape our futures, researchers come forward with inclusive strategies, designs, and applications, mindful of broadening access and striving to keep all kinds of communities included and engaged. A large body of work already exists focusing on social equity and injustices to address social imbalances [39]. Research communities such as Participatory Design (PDC) and Computer Supported Collaborative Work (CSCW), to name a few, devote much of their research efforts to further inclusiveness. In times of continuous global crisis, changes, and increasing migrations, cultural inclusion becomes a pillar to guarantee a cohesive and open future for our societies. Therefore, technology must support traversing such borders, boundaries, and barriers.

In the spirit of the ICIDS 2023 conference theme, this paper describes MEMEX, an IDS tool designed and developed to promote social inclusion through collaborative access to tangible and intangible Cultural Heritage (CH). MEMEX facilitates encounters, discussions, and interactions between communities at risk of social exclusion. It allows people to combine their fragmented experiences and memories into compelling and geolocalized storylines. Using new personalized digital content linked to the pre-existent Cultural Heritage, MEMEX allows people to tell their stories and engage with society through cultural participation. To this end, MEMEX nurtures actions that contribute to practices of recognition of differences by giving voice to individuals to promote cultural diversity. The technological embodiment of MEMEX is a smartphone app allowing non-expert users to create and visualize stories related to their memories and experiences digitally linked to the geographical locations of either an intangible (e.g., an event) or a tangible cultural place/object. The app allows users to use AR to annotate the surrounding space or any physical object or location to which their stories and memories are connected. The stories are digital images, videos, audio recordings, or textual input and can be visualized using a smartphone. Then, the users connect their experiences and memories with a specific AI tool, namely a Knowledge Graph (KG), linking CH items and places with stories bound and entangled within history. Effectively, the users of MEMEX become active actors in shaping contemporary and historical content, including new material from their experiences and memories, and personalizing cultural heritage and creative media content in a meaningful and socially inclusive manner. The target communities of the MEMEX storytelling tool are socially fragile people, sometimes blocked from participating in cultural opportunities and blocked from resources ordinarily available to members of a different group, which are fundamental to social integration. Once it reached the stage of a final working prototype, MEMEX was successfully deployed in three distinct pilot cities and communities, representing a very heterogeneous user sample particularly distinct from each other:

  • in Barcelona, community participants were migrant women, mainly working as domestic workers. Their shared experiences skewed towards women/worker rights activism and the city spaces in which they happened.

  • in Paris, community participants were inhabitants of priority neighborhoods currently in the process of urban renewal. Their shared experiences often reflected barriers (financial, cultural, etc.) of living in the neighborhood and the ongoing changes to the space.

  • in Lisbon, community participants were descendants of migrants from Portuguese ex-colonies. Their experiences reflected their heritage and its traces in the city spaces.

As we describe the MEMEX design and study results, we highlight the theme of Connectedness [7] as an empowering concept across various layers of the storytelling tool:

  • through the use of AI and automatic KG interaction to write, view, and connect stories;

  • through journeys to manually curate the trajectory of the experience in exploring the stories in the real world;

  • through AR to explore the stories as they connect to the local cultural heritage in its real-world context.

2 Related Work

Storytelling is a way to record life [11], and memories are one of the most versatile materials for stories [42]. The potential of storytelling as a crucial strategy for scaffolding meaningful experiences through time and place is long known and studied [55, 66]. Embracing Roland Barthes’s postmodern theories [4] about texts requiring different levels of the reader’s involvement (Readerly and Writerly), Interactive Digital Storytelling allows readers to be cast as both readers and authors of the same works [16, 31, 35, 56]. As technology advances, AI and AR make their way into the fabric of our everyday lives and become central features for digital storytelling tools. Designers and researchers in this field must consider how to design IDS for positive social impact. This section exposes the intersection of storytelling with AI and AR, highlighting the potential of advancing IDS to support inclusion and overcoming cultural, economic, and digital divides. The collection of studies and tools presented in this section is not meant to be exhaustive but merely to trace the backdrop of our line of investigation.

2.1 Space, Place, and Storytelling

Space has long been explored as a strategy to support storytelling and learning practices [19]. Harrison and Dourish further distinguish the notions of space and place, positing space as the structural foundation of a world. In contrast, the place is a culturally invested space, serving as a distinctive frame for spatial interaction. The two concepts are the products of social practices and jointly shape computer-mediated spatial experiences [28]. In storytelling, the concepts of space and place influence evolve the notions of Interactive Digital Storytelling, from hypertext into Sculptural Hypertext [27], from Interactive Web pages to Digital Mediascapes [59] and Location-Aware Multimedia Storytelling [46, 50] where authors and researchers design and distribute content in the 3-dimensional worlds. Moreover, in the last few decades, locative media has been deployed to champion disadvantaged neighbourhoods and communities, giving voice to the grassroots and often enhancing culture and heritage. Projects such as Urban and Social Tapestries, by Proboscis research group [36], the Media Portrait of the Liberties [54], Placewear and Storybank [22, 53] are some of the early pioneering projects that bring the power of annotating space with stories memories and content supporting activism and participation, developing a form of distributed bottom up storytelling, transforming a spatial construct into a socially relevant place. Our work builds on these efforts, learning and extending the use of technologies to include recent cutting-edge developments such as AI and AR.

2.2 AR Storytelling and Heritage

The notion of narratives as spatial experiences often takes advantage of site-specific technologies. Benford et al. [6] propose the notion of trajectory to guide the design of site-specific storytelling journeys. While authors structure the narrative’s spatial, temporal, and performative elements [52], the audience can traverse the space through spatial trajectories that result in specific experiential journeys. Extended Reality (XR), Virtual Reality (VR), Mixed and Augmented Realities (MR and AR) often feature to augment cultural heritage sites and museum visits with storytelling [1, 5, 15]. Kampa and Spierling developed a location-based AR prototype for outdoor museums [33] through which they identified and applied author requirements in designing multiple site-specific AR storytelling tools. Haahr et al. explore the space of gaming and locative stories in heritage contexts [26], highlighting the potential of AR for historical recollection. Shin and Woo [60] worked with local storytelling experts to understand the author’s motivations, goals, and needs in creating AR narratives for historic outdoor locations. Extending on the trajectories concepts of Benford, Zhang et al. [68] developed a mobile AR application prototype that guides users to the physical locations of a story using AR flags. Although much research covers digital storytelling tools engaging with site-specific Heritage, very few engage in co-designing with disadvantaged and marginalized participants [46, 49, 54]. To the best of our knowledge, this area of research still presents a rich space for investigation.

2.3 AI, KG and Storytelling Tools

In Interactive Digital Storytelling, AI techniques include (but are not limited to) decisions about characters and objects of the narrative, dramatic plotting, character agency, dialogues, and interactivity in response to players’ actions [63]. Curiosity and experimentation with AI and storytelling go back a long time. From the early 1960s experiments with the Eliza system, [64], a natural language processing computer program created to explore communication between humans and machines, Artificial Intelligence (AI) has been used to generate narrative structures, progress dramatic plots and virtual characters across both academia and industry. Research extends from the pioneering efforts of the CMU Oz Project, building interactive agent-driven Drama [41], to interactive non-linear narratives [62] and architectures [40] to balance interactive plot adaptations and character believability [13, 61, 67], to the use of AI for story generation and presentation [14]. Recently, AI has seen a rise in coupling its potential with storytelling and children’s [69] involving chat boxes, agents, and robots [9, 20]. As AI technologies evolve and get entangled with our lives [24, 45], decisions on how these technologies support us are made by those with the knowledge of how to develop them (e.g., engineers); most end users struggle to understand how AI supports them and have no influence on the design of such digital tools [2]. Lately, HCI research has strived to include AI in a human-centered approach and to empower creative processes [32]. The latest AI Natural Language Processes (NLP) models (such as Chat GPT) are opening up possibilities and inspiring authors and audiences to experiment with what could be the future of interactive narratives [10]. ChatGPT is an AI-powered chatbot that uses NLP to generate original story ideas. While ChatGPT is not a writing tool, and it is not going to write stories in place of the author, it can help create ideas. AI responses can be used as a jumping-off point for story writing. By understanding how a narrative AI system behaves, authors can benefit from the generative capabilities of AI systems to generate the stories they wish to tell [63]. Therefore, tools become partners, not only manipulable but responsive and supportive to the user’s workflow [25]. As pointed out by recent research, future work in this area should look at how AI can inspire and empower users to write the stories they want [12, 23, 37]. Finally, aligning AI and crowd-generated content allows recasting the creative process as “reading-writing” [17], where a user can influence and be influenced in a collaborative partnership with digital tools [2].

2.4 Summary

In summary, IDS research combining storytelling with life stories, locative technologies, AI, and AR, is abundant. IDS tools have often been deployed to support cultural heritage, sometimes covering disadvantaged areas and serving marginalized communities. Nevertheless, very little has been (co-)designed directly with and to engage with those inhabiting the fringe of society, exposing them to cutting-edge technologies such as AI and AR. MEMEX covers this ground, promoting the use of AI and AR as storytelling tools to support marginalized communities to bond, share, and participate and, in doing so, traverse borders, boundaries, and barriers. MEMEX approaches this issue by harnessing the user’s experiences through AI-supported authorship and site-specific AR storytelling. Moreover, it contributes to widening audiences’ inclusive participation in Cultural Heritage by diversifying and enriching digital content for Cultural Heritage. It identifies and supports authors’ motivations to engage with the sites as a creative resource, enhancing their socio-cultural significance [21].

3 MEMEX Storytelling Tool

The MEMEX EU project, of which the MEMEX IDS tool is the output, comprises a technological and a societal concept. MEMEX technology is instrumental in achieving the project’s societal objectives. In the scope of this paper, we describe the technological concept and the digital storytelling tool in detail, concluding with the results of a study conducted at three pilot destinations involving users from three examples of communities at risk of exclusion.

3.1 Overview of the Tool

MEMEX considers storytelling a stepping stone for cultural inclusion. Therefore, the resulting MEMEX storytelling application is intended to:

  • Gather memories of communities at risk of exclusion, connecting these with physical places, locations, and objects to promote social cohesion.

  • Create assisted augmented reality experiences in the form of stories that intertwine the memories of participating communities.

  • Develop techniques to semi-automatically link images to a location and connect to a new open-source knowledge graph to facilitate assisted storytelling.

Prominent in the MEMEX tool is the concept of “connectedness”, which emerged during co-design with the communities at risk [49]. It is used as a common unifying thread among these different goals: connecting stories to physical environments, to virtual environments, and to information (external to the app and within the app). “Connectedness” is at the heart of the storytelling activity enabled by the tool, connecting authors among themselves and their audiences, as well as their stories and memories with the surrounding heritage. Finally, “connectedness” extends to MEMEX as a socio-technical system, as it requires understanding social structures, roles, and rights to derive hardware, software, personal, and community requirements. Regular activities involving the participation of various combinations of stakeholders took place during the ideation, design, and development of the application to achieve a consensus between the different consortium members and co-design parts of its features.

3.2 Basic Interactions, Story Writing, and Viewing

The tool’s story writing and viewing aspects were developed over three iterations of user feedback moving towards a mature state. These functionalities include a guest option and authenticated users for authors. A story authoring interface (as shown in Fig. 1a) facilitating authoring with a set of inputs to structure stories: i) cover image; ii) story title; iii) story body text (optionally split into sections) with WYSIWYG interface; iv) linked cultural content suggestions (see Sect. 3.3); v) geographic location; and vi) a set of tags (from an expert-curated list). The interface then provides viewing of the stories as a list or on a map.

3.3 Connectedness Thread Embodied in the Tool

The MEMEX storytelling tool embraces the notion of connectedness at various levels. At a macro level, two different dimensions of connectedness come together: firstly, regarding interlinking cultural Heritage content with personal stories and memories of the marginalized community participants (Content Connectedness); and secondly, the tool connects different locations geospatial through journeys within physical space, taking the audience from one story (location) to another, following threads of memories or cultural topics (Geographical connectedness).

A graph structure provides the realization of both kinds of connectedness with content provided by WikiData [18]. We follow the approach of [43] for constructing a Knowledge Graph (KG) to provide content suggestions in the form of textual information and images [44].

Content Connectedness: Content can be connected in several ways, from during the authoring processing to post-processing using AI techniques. The KG can be searched for textual and image suggestions to support the authoring process. We frame this as an explicit process providing semi-curated content. The user can then add content to the stories, intertwining with their narrative. The High-fidelity wire-frames for the content suggestion process are shown in Fig. 1.

Fig. 1.
figure 1

Hi-Fi Content Suggestion Design: We design a workflow from a story (a) facilitating search within the Knowledge Graph for both (b) intangible details (c) images of objects and places as a slideshow (d) and included in the story (e).

Full size image

Moreover, stories are associated with meaningful tags that connect each story to one or more common themes. Common themes within stories are, therefore, automatically connected. The tags provide one way of interlinking stories based on expert-curated topics. However, they lack fine-grained details of the specific elements of the story. Therefore, we use Natural Language Processing (NLP) to enhance story connectivity.

We utilize the Named Entity Recognition (NER) technique to extract important keywords from a story. More precisely, we use SpaCy [30] with wide linguistic features. SpaCy annotates the story text with different types of named entities. We focus on the following types: i) Geopolitical entity (GPE), i.e., countries, cities, states; ii) Non-GPE locations, mountain ranges, bodies of water; iii) Buildings, airports, highways, bridges, etc.; iv) People, including fictional ones.

Once a user has completed and submitted for publishing a story, the MEMEX platform then performs the following steps:

  1. 1.

    NER tool extracts named entities (i.e., keywords), filtered by language, from the story text.

  2. 2.

    Where there is a match between an extracted named entity and the KG entities’ labels, a link named “related to” will be added between the story and the KG entity.

  3. 3.

    Where the story has at least one common KG entity or common keyword with another published story, a link named “similar to” will be created between the two stories.

Geographical Connectedness (Journeys): In the context of MEMEX we define a journey as a set of several stories that form a geo-spatial narrative. We consider that a journey can be either sequential, allowing a path through the space, or unordered, allowing starting at any point. Each journey has a title, a description, and a list of stories, where the stories included in the journey may or may not belong to the user creating the journey. Each story in the set has an optional text component that the user can use to describe why the story was included in the journey. A user creates a journey by choosing a story as a starting point and then sequentially adds stories (irrespective of whether the journey is ordered). In addition, the journey creation and editing interface allows for configuring several graphical parameters of how the journey appears for the viewer (e.g., color). The process of the journey creation is shown in Fig. 2.

Fig. 2.
figure 2

Hi-Fi Journey Design: (a) Journey creation interface; (b) Selection of stories to be included in the journey; (c) list of journeys; (d) Visualisation of an ordered journey; (e) visualization of the unordered journey.

Full size image

3.4 Bringing Connected Storytelling into Augmented Reality

A simple but scalable approach to AR content brings the stories and tangible content into reality. During the setup phase, virtual markers of the MEMEX icon (as shown in Fig. 3) are placed within environments. Each environment is created using Google AR CoreFootnote 1 to position an (easily recognizable location) and virtual element(s) in the scene, which is linked to nodes within the KG. This article focuses more on visualization, as the creation can be approached in many ways with the rapid acceleration of AR technologies.

Given the graph structure within the KG, we would like to express the connectedness within the finite space of a mobile display while remaining usable for users. Displaying graph data within AR is a fleetingly addressed problem, with most approaches focusing on displaying a whole graph [38] or complex full-screen navigation around the graph as a content management system [34]. Displaying our large KG is not plausible, and complex UIs result in a loss of immersion, so we opt to display only immediate connections to the node to which the virtual element in AR is connected. We design an interface where once a virtual element is selected (Fig. 3b), a cloud of connections is displayed as icons in anti-clockwise ordering (Fig. 1c). The user can then click on either the central virtual element or any of its surrounding icons to view more information or, in the case of stories, drop out of AR to read the story in a standard 2D display (Fig. 3d). We opt for this approach as visualizing a story as a virtual text in AR is cumbersome to read, and also, reading stories takes time, which will have significant implications on battery and device temperature for a task that does not benefit from being in AR.

Fig. 3.
figure 3

Hi-Fi AR: (a) Place marker in location; (b) Object marker on a monument (c) Expanded marker showing connectivity to tangible and intangible content as well as stories (script) (d) A story visualized as a 2D interface for comfortable reading.

Full size image

4 Study

Local NGOs associated with the project recruited participants from communities at risk of social exclusion in three pilot cities (Barcelona, Paris, and Lisbon). The participants’ engagement was motivated by exploring public heritage through storytelling. It took multiple forms and iterations, which were the focus of other publications (exploration of needs and requirements [49], the creation of stories about their experience of living in these cities [47], co-creation and evaluation of prototypes [48]). The stories produced by the participants were geolocated multimedia stories (with text, photos, videos, and audio) and connected to the heritage of each pilot city.

In this section, we report on an “in the wild” study across the three pilot locations (N = 15) to test the third and last version of the MEMEX prototype in the locations where the marginalized communities live and provide stories about it. The study is designed to capture the user’s experience with the tool through the Attrakdiff scale [29] and probes participants’ expectations for the system, how they would change it, and how it would fit into their lives or workflow through a final semi-structured interview. The authors of this article developed the prototype, the study protocol, and the analysis; local NGOs were tasked with recruiting participants, setting up the AR experience, and evaluating the prototype (following the established protocol).

Participants Demographics: The recruitment for this study included community participants (who did and did not participate in previous activities), cultural operators stakeholders (such as museum and cultural professionals) and community operators (local social partners and NGOs), as well as some general audiences (who live or are visiting the pilot area). Of the 15 participants involved, three were from Barcelona, seven were from Paris, and five were from Lisbon. The sample was equally distributed in gender and typology; most participants were between 25 and 34. While most participants were experienced with smartphones, most reported no or very little experience with AR applications. To preserve anonymity, participants are identified by the first letter of the pilot city and a total number (e.g., P1 is Participant 1 in Paris, and B3 is Participant 3 in Barcelona).

Setup: The local social partners of the project selected two to three representative stories (from a corpus created earlier in the project during several digital storytelling workshops). With the help of technical experts, the Social partners situated the story content in real work by manually placing some AR anchors. The local social partners verified the integrity of the Knowledge Graph (KG) information related to the chosen area. They curated the creation of journeys, which manifested as specific paths through the corpus of stories, connecting stories.

Procedure: Before the experience, participants signed an informed consent form and filled out their demographic data (including previous experience with smartphones and AR applications). A local partner facilitated the experience at the location by providing the participant with a mobile with the prototype installed and a list of tasks. These tasks included: (1) exploring the physical surrounding location to find the story content; (2) visualizing the story as Augmented Reality media in the space where it is anchored; (3) visualizing the information provided by the KG and connecting to the Heritage connected to the story out to the physical location; and (4) use the journey feature by exploring the vicinity to find other stories which are connected to the first one visualized.

During the study, facilitators took notes and observed participants’ interaction with the prototype. After the experience, participants filled out a form with the AttrakDiff scale, a tool designed to gauge the user experience through usability and design of an interactive product [29]. When no validated scale translations existed (e.g., Spanish), local partners translated the scale to the users on the fly. The scale has 28 word pair items that can be combined into 4 component dimensions:

  • Pragmatic quality refers to the prototype’s usability (ability to achieve tasks).

  • Hedonic stimulation quality refers to the prototype’s potential to provide a positive emotional experience.

  • Hedonic identification quality refers to the prototype’s potential to provide ownership (e.g., a social function or self-identification).

  • Attractiveness quality refers to the overall attractiveness of the prototype based on pragmatic and hedonic qualities.

Finally, the facilitators conducted a short semi-structured interview with participants (audio recorded, transcribed, and translated by native speakers). The interview included questions about the experience of using MEMEX, and changes desired/needed, awareness of locations, and how data was connected to the physical location.

Analysis: The quantitative data from the AttrakDiff scale was analyzed in R [58], with figures produced using the ggplot2 package [65]. Analysis of qualitative data was conducted using thematic analysis [8] with NVivo (1.6.2),Footnote 2 a qualitative data analysis package.

4.1 Results

Fig. 4.
figure 4

Average values for each AttrakDiff word-pair item. From top to bottom: individual values per pilot location and combined values.

Full size image
Fig. 5.
figure 5

Average values for AttrakDiff dimensions. From left to right: combined values and individual values per pilot location.

Full size image
Fig. 6.
figure 6

Portfolio-presentation with average values of the Pragmatic Quality (PQ) and Hedonic Quality (HQ) dimensions, and confidence rectangle. From left to right: combined values and individual values per pilot location.

Full size image

The analysis of the quantitative data from the AttrakDiff scale positions the user’s experience of the prototype in the average scoring range for all four items of the scale (Pragmatic quality or Usability; Positive emotional experience or Hedonic stimulation quality; Ownership or Hedonic identification quality: and Attractiveness). The results are described and represented through three different visualizations below:

  • The usability of the product. MEMEX usability (Pragmatic quality) scored average values according to the AttrakDiff word-pair items. See Fig. 4. Values between -3 and -2 are critiques needing improvement; values between 2 and 3 are strong points of the system. Average values for the MEMEX prototype are between -1 and 1, indicating no pressing issues exist, but there is space for improvement. In particular, lower scores are more prevalent for items connected to pragmatic quality, indicating usability issues in using the prototype in public.

  • Positive emotional response and ownership. MEMEX scored average values for its ability to trigger positive emotional responses and ownership of the tool (Hedonic stimulation and identification qualities). See Fig. 5. Values between -3 and 0 are negative opinions of the system; values between 0 and 1 are standard opinions of the system and values between 1 and 3 are positive opinions of the system. While the values differ slightly for each pilot community, the combined scores for all the pilots across dimensions are between 0 and 1, representing a perception of the prototype in the standard range.

  • Attractiveness. Based on the combination of the values from the pragmatic and hedonic quality, the attractiveness of the prototype is centrally placed in the scoring system, represented through a rectangle. See Fig. 6. The center of the rectangle represents the mean value, while the rectangle area represents the confidence level. For MEMEX, scores place our prototype in the “neutral” sector.

From the interviews, through the thematic analysis, researchers highlighted the following themes:

  • Novelty of the tool - Several participants praised the novelty of using AR aligned with heritage (e.g., P1 “This is the first time I have used augmented reality, in this context it allows the discovery of places/histories in the neighborhood based on heritage elements. AR allows the territory to be highlighted.”, P2 “An enjoyable experience, it’s fun to work with augmented reality [...] a great idea to discover areas”, L2 “I thought it was innovative, this thing of augmented reality. It’s not that it is more interesting, but it is cool. It’s creative.”). Lack of experience with the technology was not a hindrance to enjoying it as they quickly learn how to use it (e.g., L5 “Once you start using it, you get used to the app’s language. There is a first barrier from being something new that doesn’t come with an instruction manual, but once you enter with something, it becomes easy.”).

  • Valuing the sharing of personal stories - Several participants praised the prototype’s connection to personal stories (e.g., L5 “[...] attempt to create a bridge between the history of the place and the history of someone connected to the place.”, L2 “I think that if we see many people doing it, I would probably spend an afternoon in Lisbon checking where there are stories.”).

  • Valuing the KG in providing extra information about CH - Regarding the heritage-related information provided by the KG, participants cherished the connections made by the AI-supported system (e.g., L1 “What I liked is the knowledge and the historical background about the place that I can retrieve from the MEMEX application (enabled by the KG feature).”, L2 “One thing that would be cool, aside from the story the person has written, is to know historical facts about the building or monument.”); participants also point out the need to further curate the information provided by the KG, as sometimes the reported information is very basic, sometimes banal, or too convoluted to make sense of (e.g., P7 “[...] not very interesting information, hard to understand”, L4 “[...] I’m a local, so it’s mostly information I already know about the places. I didn’t want to read because it seemed too much text. I think the information could be (delivered) through a more visual language”. Such reactions indicate a need for granularity and different levels of detail depending on the use case, as residents know greater detail than what the MEMEX can provide. However, such participants were not the overall target of the project.

  • Navigation of the physical space - Participants described navigating the space with the MEMEX prototype in hand as cognitively overloading. They expressed the need for more balance between the app’s demands and the surrounding space. They were aware of what was happening around them and unaware of how the app would work (e.g., L5 “[...] it was a compromise. I was alert because this plaza was lively, with many people and a lot of movement, and I was here. But at the same time, I was in the plaza and focused on how this app works. So it was a compromise between being present while understanding how this is operated”). Resonating with many locative media experiences literature [50], the app screen requiring too much focus, and the action of pointing the phone to locate content [51] led to uncomfortable interactions in the space (e.g., L4 “I wasn’t very conscious, I was too much in my bubble. I even forgot I was in a tourist place. Amid it, I even touched my bag to check if I had everything [because of potential theft] but I was looking at the screen the whole time. I only looked at the area through the screen. And suddenly, I was on top of people and they looked into the camera, which is a bit invasive, and I realized I was pointing to people and it wasn’t supposed to. So, no, I was not aware at all.”). The tension of striving for immersion while remaining sufficiently present is a fine line and proved to be a mixed experience and highly dependent on the participant’s background.

  • Usability issues - The study highlighted several usability issues with the prototype. As is common with AR experience, the participants sometimes had difficulty locating AR content in space. They expected the MEMEX prototype to facilitate the exploration of the space and the location of the markers by providing more information on the map through vibrations or visual feedback. Participants also reported issues in understanding the icons related to the KG. Highlighting the challenge of the finite space and clear communication of meaning. Naturally, this problem subsequently affected the use and testing of the KG feature in-depth, as some users could not figure out how to engage with the feature. In addition, internet connectivity proved challenging as the responsiveness and speed of the prototype (particularly in loading and playing videos were often a source of frustration and complaint. However, such could be overcome by local caching of the data.

5 Conclusions, Limitations and Future Work

While location-based narratives are not new topics for IDS researchers and practitioners, when brought to the margins of society, such interventions are still praised for their novelty, particularly in conjunction with cutting-edge technologies such as AI and AR. The AI future of the Knowledge Graph, supporting authoring and connecting stories to the surrounding CH, was perceived as potentially helpful and exciting but needing customization to individual users in usability and content curation. The combination of AR with the navigation of the surrounding space, augmenting the space with personal stories and heritage information, was well received. Nevertheless, users accused a certain lack of balance of attention between the application and the surrounding space, echoing much locative media literature. The opportunity to connect with others and share personal stories was one of the most celebrated features that overcame the usability challenges and lack of familiarity with the new technologies. Overall, results confirm that there is much value in bringing cutting-edge storytelling tools to the margins, exposing a wide variety of public to viewing, sharing, and authoring tasks, and wielding together their stories and memories with existing cultural discourses. MEMEX can be seen as a successful example of technology connecting people, places, and experiences to traverse borders, boundaries, and barriers, with exciting challenges still open for research investigation.

Limitations and Future Work. We are aware of several limitations of this work. More consistent and wider testing is needed to confirm its usefulness and appreciation from the communities at the margins. As the testing comprised a mixture of stakeholders, focusing on the single communities can highlight useful customizable features within the app. Specifically, it can be seen that participants with a rich knowledge of the local area require more detailed information to be presented than was provided within MEMEX, while visitors to the area can be satisfied with higher levels of overviews and historical details. Moreover, the participants’ age range and technology literacy can also be harnessed to personalize app features. Finally, as the project ended, the iterative development of the tool ended with it. Using the tools within other research contexts can provide opportunities for further iterations of the prototype, improve usability, and incorporate user suggestions. In fact, after its final deployment, the MEMEX app has been adopted by two EU-funded projectsFootnote 3 geared towards supporting citizens inclusion with digital tools in the face of climate change and sustainability, social and ecological alike. The tool will be used as a test bed for further explorations of storytelling in the context of marginalized communities, such as refugees, thanks to the interest and involvement of the Portuguese Association for Refugees (CPR) in continuing to provide support and context for further research. The Italian Institute of Technology, in partnership with Genova municipality, is exploring the use of MEMEX to co-create with local communities and non-humans whose voices within the community of the fishermen of Vernazzola village in the context of the Bauhaus of the Seas - New European Bauhaus (NEB) lighthouse project.