Keywords

1 Introduction

The LabInVirtuo project aims to develop and validate, Realistic Sensory Intelligent Virtual Environments (RSIVE). These environments are envisioned as cross-disciplinary virtual laboratories, in which various players may work cooperatively on projects related to culture and heritage.

The first hypothesis of this project is that the immersion of users in a Realistic Intelligent Sensory Virtual Environment, where bodies are kinematically, gesturally, and sensorially involved, allows for the following:

  1. (1)

    An increased retrieval of procedural, episodic and autobiographical memories that are usually interrogated or worked on when producing oral accounts traditionally.

  2. (2)

    Capturing gestures and embodied knowledge in context.

  3. (3)

    Memorization of knowledge.

Our second hypothesis is that collaborative mixed reality allows for new methods of cross-disciplinary research and cultural facilitation, with a qualitative and quantitative leap with regard to eliciting and restituting knowledge and know how.

2 Context

LabInVirtuo strives to preserve industrial occupations and know how. It is cross-disciplinary, bringing together people with backgrounds in humanities and social sciences, and computer and mixed reality research scientists.

2.1 Positioning

Today, digital technology to capture our heritage is in established usage for all professionals in the field. Projects carried out within institutions display heritage items in their historical context [1] and allow for their virtual manipulation [2]. To our knowledge, none of them lets them be operated so that their usage may be archived. Yet VR seems to have a significant effect on the acquisition of knowledge about these historical objects [3]. According to Barreau [4] “In the more distant future, the stakes will lie in the simulation of, and interaction with, virtual humans from the past”.

LabInVirtuo focuses on human activity, and the elicitation and restitution of expert knowledge in an RSIVE. It complements current approaches focused on the technical object, as in the issues raised with regard to the Digital Heritage Reference Model (DHRM) [5]. Being interdisciplinary, the project is at a crossroads of various fields: history, anthropology, industrial heritage, virtual and augmented reality, coupled with a sensory aspect, which, while a little explored field [6,7,8], is essential to achieve a better virtual restitution of usage and embodied knowledge, given all the data this aspect carries.

Clarifying knowledge in virtual environments is the purpose of Intelligent Virtual Environments (IVEs) [9]. Some works in the VR field [10, 11] come close to the objective of designing IVEs using metamodels. These models only allow the end user to access knowledge, not to create new knowledge in real time. Embodied Conversational Agents (ECAs) are increasingly frequent in IVEs, taking on the roles of helpers, tutors, or attentive psychologists [12] and producing clear benefits [13, 14]. For interaction to be effective and satisfying, an ECA must be able to speak but also listen, modify, and adapt its behavior in accordance with that of the user [15]. However, the agent behaviors developed in this domain focus on the feedback exchanged between participants while speaking, but rarely on the semantic content of the messages or in the agent’s knowledge base. Recent works [16] describe a conversation system, involving social robots or conversational agents, based on the semantic content of speech. This system aims to generate rewarding interactions through the restitution of subject-appropriate knowledge, but does not provide a way to update this knowledge. For our project, we want ECAs to serve as interfaces for expert users to update knowledge or enter new data.

From a methodological point of view, developing an RSIVE raises the question of how to convey realistic, sensory information from the historical data that was collected:

  1. (1)

    How to restore a past industrial setting in all its dimensions: tangible and in-tangible; from material traces, which include old preserved workshops, usually with their machines removed; from archival sources; from oral memory…? This question implies enriching the existing reference works [17, 18] with notions of authenticity and integrity, and tackling the challenges identified twenty years after the Nara document [19] with the acknowledgment of “the emergence of new forms of heritage that have not previously been given much attention”. […] as “emerging modes and technologies for accessing and experiencing heritage, which may include new ways of enhancing or providing perceptions of heritage through increasingly widespread practices such as historical role play, living his-tory, historical reenactments, computer games set in the past, and the creation of virtual realities augmenting visitor perceptions”.

  2. (2)

    By extension, and given the prospect of immersion in an RSIVE, sensory studies are necessarily central to our reflection. Corbin [20] suggests that there is perpetual tension between the senses known as “social” (sight and hearing) and the rest (touch, taste, and smell) which “provide the feeling of objects” and “give information on things’ true natures”. Additionally, “Means of doing and means of feeling cannot be dissociated if we want to apprehend, in their entireties, the bodily tactics which intervene in certain situations of shared cognition aiming at the elaboration and the collective exploitation of knowledge” [21]. Let us also quote Gélard [22]: “the senses have been slowly but securely permeating research settings and they are opening up many new prospects. Through their ability to express specific information, the senses inform us about the societies within which the anthropologist examines them. The senses are identified by and in singular manifestations (visual, auditory, olfactory, etc.) that we must describe and decode.” […] “It is under this methodological angle of a decryption of the senses and their diversity that the most fertile anthropological approaches are situated”.

Although the question of sensory landscapes has been studied from the perspective of Humanities and Social Sciences (HSS) [23], a cross-disciplinary approach to the restitution of multi-sensory landscapes from the perspective of Digital Humanities remains to be conducted. These aspects will therefore be analyzed through the various senses that humans develop in a technical context, from hearing (soundscape archaeology) to touch (tactile and haptic), from smell to thermoception and, of course, sight. While graphical representations are a staple of 3D modeling projects, the remaining senses are more rarely called upon, even though they fully contribute to understanding a complex technical system, through the interpretable information they provide.

The originality of our interdisciplinary approach lies in:

  • The interdisciplinary coupling of previous and ongoing work in the present consortium including the alignment of the HSS activity metamodel ANY-ARTEFACT [24, 25] (and its associated history methodology) with the DHRM metamodel [26, 27] and the MASCARET metamodel in IVEs [28] (and its IVE design methodology used in research and industry).

  • The hypothesis that situating users in a collaborative IVE where the body is engaged from a sensory, kinematic, and gestural perspective results in:

    1. (a)

      An increased retrieval of the different procedural, episodic and autobiographical memories that are usually interrogated in the classical setting.

    2. (b)

      More efficient elicitation of expert knowledge. LabInVirtuo constitutes a mode of implementation and evaluation of this hypothesis.

  • New modes of cultural facilitation:

    1. (a)

      Reaching a sensory informed environment, which kinematically and physically involves its users.

    2. (b)

      Zooming in and out according to multi-scale space–time modalities.

  • The ambition to validate methodologies of preventive preservation of cultural heritage and industrial skills, considered as factors of innovation to build the society of tomorrow.

2.2 Use Cases

Our model needs to be generic enough to be applied to various use cases in four French cities:

  1. (1)

    In Brest, the Pont National, inaugurated in 1861 and destroyed in 1944, and the Pontaniou forges in the arsenal, which house a Schneider & Compagnie power hammer from 1867.

  2. (2)

    In Nantes, Cap 44, a former flour mill built in 1895 and transformed into a storage warehouse and then into an office building in the 1970s. Its interest lies in being one of the first large buildings made of reinforced concrete in accordance with the system designed by François Hennebique.

  3. (3)

    In Paris, the Numasurf, a numerically controlled milling machine, the only remnant of the UNISURF system, and the first system allowing end-to-end digital design, which was designed by Pierre Bézier for Régie Renault in the late 1960s.

  4. (4)

    In Belfort, Techn’Hom, formerly made up of several industrial sites in the textile and mechanical construction sectors such as DMC or Alstom and which, at the beginning of the twenty-first century, became an economic business park spanning more than 100 acres.

Each use case needs to be formally described through an ontology (the digital corpus) and reconstituted as a virtual environment. To this end, various kinds of sources are gathered: three-dimensional scans, floor plans and texts, among others.

While users are involved in the virtual environment, the digital corpus can be interacted with in three manners (Fig. 1):

Fig. 1
An illustration represents the phases of data collection, working or exchange, and data restitution. The avatars in the data collection phase are converted into the embodied conversational agent.

Three ways of interacting with the digital corpus in LabInVirtuo (Modeling Virtualys, diagram: R. Querrec)

Full size image
  1. (1)

    Collecting data in the virtual environment by interacting with the reconstituted industrial system.

  2. (2)

    Working and exchanging: we could compare corpus built by experts in different fields, e.g. historians and archaeologists. With the same collaboration mindset, we can present reconstitution ideas within LabInVirtuo to have them validated.

  3. (3)

    Restituting knowledge in facilitation and teaching contexts.

3 Proposed Models

The goal of the project is to design the tools and interactions that will enable a bidirectional link between the digital corpus and the virtual environment (Fig. 2).

Fig. 2
An illustration of the interaction between the digital corpus and the virtual environment. The virtual environment has a flow chart that starts with the entity and goes through entity class, class, and instance specification.

The global architecture of LabInVirtuo, exemplifying the bidirectional relationship between the virtual environment and the digital corpus (Modelling Virtualys, Diagram R. Querrec)

Full size image

3.1 The Digital Corpus

The digital corpus comprises not only the historical data and sources as ontologies, but also the technical provisions to preserve and access them. The ontologies are stored in Omeka S with ANY-ARTEFACT. ANY-ARTEFACT is a generic ontology which adapts CIDOC-CRM for defining specific ontologies in science and technology. ANY-ARTEFACT is designed around the following concepts (Fig. 3):

Fig. 3
A triangle labeled activity denotes actors, artifacts, and knowledge in the vertices.

The main concepts of ANY-ARTEFACT (Diagram M. M. Abiven)

Full size image
  1. (1)

    The concept of Actor, which can be:

    1. (a)

      A human being as a person.

    2. (b)

      A group, i.e. two individuals or more arranged in a social structure.

    3. (c)

      A character, representing a status or an appointed position (for instance a blacksmith).

  2. (2)

    The concept of Artefact, namely a human-made production with a function, that is, the ability to perform an action, as well as a use.

  3. (3)

    The concept of Knowledge: specifically theoretical and procedural knowledge.

  4. (4)

    The concept of Activity: an activity is envisioned as a series of (unique or repeated) events involving actors, artefacts and a knowledge system. An activity unfolds as time passes and is spatially situated [23]. In the context of ANY-ARTEFACT, we call industrial cultural practices any kind of industrial activity:

    1. (a)

      Periodic industrial activities (spatially and temporally situated, with a set duration), which we call technological or scientific procedures, involving actors, artefacts, and knowledge (for example forging a metal part).

    2. (b)

      Unique events (the creation of a factory, a technical failure, an accident, etc.).

Therefore, an industrial cultural practice is fixed at a set time and place (e.g. all the activities carried out at the Pontaniou forges in the nineteenth century within the Brest Arsenal). A procedure is an operational sequence that will be regarded as identical over the related period (e.g. the anchor forging procedure in Brest between 1710 and 1760). Hence, it happens repeatedly. Unique events are samples of such procedures (the forging of an anchor on 16 January 1730, on 6 February 1755, etc.). These specific ontologies match the four use cases we previously described.

On a technical level, we created a library to interact with Omeka S from the virtual environment. This allows us not only to look up concepts and sources, but also create new ones in real time. Storing historical data in a tool like Omeka S is useful in that it allows historians to keep their usual data collection and facilitation workflows. However, we posit that the VE constitutes a more natural interface for browsing, simulating and extending this digital corpus. Designing these virtual environments requires resolving two scientific obstacles.

The first obstacle is implementing the methods to establish the bidirectional link between the digital corpus and the virtual environment by offering multi-sensory interactions.

3.2 The Virtual Environment

The informed virtual environment must be generated from the digital corpus data, and we need to propose interaction metaphors for the VE. To this end, we rely on an informed virtual environment metamodel: MASCARET. MASCARET [28] is based on the UML metamodel and is used to describe technical systems and the human activities related to them. The operational semantics is specified, which allows for interpreting MASCARET-described models in virtual environments.

The main added value of MASCARET in this context, compared with ontologies, is that it has a precise operational semantics for activities (Fig. 4).

Fig. 4
A block diagram represents the organizational entity, activity, role, and resource under the organizational structure. It indicates resource assignment for the entity and role assignment for the agent, under the organizational entity.

The concept of Activity in MASCARET (Diagram R. Querrec)

Full size image

An activity is declared within the framework of an Organizational Strucure, which also references roles and resources. The activity is actually performed by an organizational entity that assigns resources to entities and roles to agents. Thus, Entities correspond to the notion of Artefact in ANY-ARTEFACT-O. In a similar way to ANY-ARTEFACT-O, Entities have Properties, which can be behavioral or structural. As MASCARET is based on UML, it also includes the notion of class (Entity Class).

When reading the ontology, if an artefact matches an instance, MASCARET will then create an Entity. If it represents a concept in the ontology, then an Entity Class will be created in MASCARET. An Embodied Agent in MASCARET is an agent, which means it has its own autonomous behaviors and its own properties. It also possesses a body by way of an Entity.

Algorithms relying on the alignment between ANY-ARTEFACT and MASCARET have also been produced. On a first level, we consider 3D, sound, gestural, haptic and tangible data, and if possible smells and tastes. On a second level, we look at how to manage human activity within the virtual environment. An activity may be, for instance, a series of actions undertaken by machine operators with the goal of manipulating a technical system. This level relies heavily on the first one, because operations may alter the geometry (position, orientation) of part of the system, generate sounds, or necessitate operators (either autonomous agents or LabInVirtuo users) to perform gestures. The main originality of this task will lie in the ability to enrich and modify the digital corpus from the virtual environment. For the first level (3D objects, sounds…) altering raw data from the VE is not necessarily relevant; however, this becomes relevant in the case of semantic data. Conversely, describing activities, operations and gestures in the virtual environment through demonstration is more pertinent.

3.3 Embodied Conversational Agents

The second obstacle lies in designing interactions with Embodied Conversational Agents (ECAs) and VEs. ECAs are meant to act as interfaces between users and the knowledge in the informed virtual environment, extracted from or bound for the digital corpus. The first goal is to provide ECAs with the ability to understand questions and information expressed by the user and to verbalize answers and information. The ECA’s verbalizations will be based on classical text-to-speech techniques, but their content will arise from the ECA’s reasoning about the knowledge base of the informed virtual environment. Our objective is not to work on automatic natural language processing. Here, the originality is we can devise a grammar based on the metamodel used to design the informed virtual environment. The second goal is to design the algorithm ECAs will use to reason about the knowledge base and user interactions within the VE. The aim is to offer new modes of interaction (haptic, gestural, tangible…) to query or inform the knowledge base through agents. Agents must also be able to reason to generate new interactions with the users. For instance, An ECA must be able to answer users’ questions, pinpoint deficiencies in the knowledge base (such as a 3D object with no semantic data) or inconsistencies (an item with different names given by different users). From these interactions, new knowledge will emerge, and this knowledge will be relayed to the digital corpus.

4 Data Restitution

Designing facilitation situations in virtual environments designed according to the proposed method involves three roles whose relationship is presented on Fig. 5.

Fig. 5
An illustration depicts the historical data going through any artifact, followed by the history expert and facilitation expert. The facilitator presents the facilitation scenario.

The organization of roles to develop facilitation scenarios with the LabInVirtuo method (Diagram R. Querrec, M. M. Abiven)

Full size image
  1. (1)

    The history expert is a specialist in the research field and has experience with the notion of industrial cultural landscapes. From the knowledge in the field about a given cultural landscape and ANY-ARTEFACT meta-ontology, they create a specific ontology for their field which will be automatically aligned with MASCARET.

  2. (2)

    The facilitation expert has high levels of expertise in history and digital humanities. They define generic facilitation actions that will serve to guide or correct the user within the environment.

  3. (3)

    The facilitator: from the data provided by the history and facilitation experts, they design facilitation scenarios, that is, situations in which the user will perform actions and interact with items in the environment. These scenarios involve the acquisition of knowledge or procedures and are adapted to fit different audiences.

5 Conclusion and Perspectives

We consider the industrial landscapes represented by the use cases as sensory landscapes that solicit different senses. By sensory landscape, we mean an environment where the senses are solicited either in terms of bodily context or by participating in the industrial activity through the information they provide.

In this project, we would like to continue the work within this framework by trying to represent in the environments the sensations felt by former users of the systems. Currently, the immersion of the user in a virtual environment and the possibility of carrying out actions solicit a certain number of senses (sight, hearing). To make this immersion as complete as possible, reflections are engaged in order to solicit other senses such as touch and smell. A more advanced study of sensoriality would allow us to get as close as possible to these industrial landscapes and thus, to express specific information that informs us about society [22].

When it comes to cultural heritage, two notions are essential: those of authenticity and integrity. Authenticity corresponds to the knowledge and understanding of cultural heritage, but also to its meaning [29]. This authenticity is ensured by verifying the sources of information about its relevant values as specified in the Nara Document on Authenticity (1994) which represents a key reference for the recognition, conservation and restoration of cultural heritage. The conditions of integrity, on the other hand, require judging the extent to which the property possesses all the elements necessary for the expression of its value and that it offers a complete representation of the characteristics that convey the property’s meaning. In this project, we would like to be able to express information related to these two notions. Indeed, authenticity and integrity data must be represented in ontologies by historians. These expressed data aim at guaranteeing the scientific reliability of this cultural heritage, from the collection of the historical data to the presentation of the RSIVE to the final public.

We plan to undertake two distinct experiments to test our initial hypotheses. One is a laboratory experiment, during which we will compare ontologies created while immersed in LabInVirtuo with ones built using traditional methods (i.e. by describing them in Protege or OntoMe). The other experiment will be conducted in a museum setting, as a temporary exhibition. Visitors will be presented with a facilitation scenario using LabInVirtuo. We will collect qualitative data about their memorizing level and understanding of the subject, to compare LabInVirtuo with more traditional facilitation methods.