1 Overview

Architecture, Engineering, and Construction (AEC) are advancing to respond to a new set of standard and code demands: Buildings must be more sustainable, maintainable, accessible, responsive to user needs, resilient, and safer. As such, it is critical to prepare a new generation of professionals ready to meet these needs, as well as to enable the broader community to participate in the management of the built environment. Public access to meaningful data from buildings will impart potentially disruptive, long lasting, and transformative effects on learning and scholarship engagement in Science, Technology, Engineering, and Math (STEM) [1]. In the long term this would facilitate decision-making, participation and trust in science and engineering [2].

While SMART buildings (buildings with integrated structural sensors [3]) help to evaluate and maintain the built environment, an increasing number of facilities also attempt to promote sustainable engineering practices by making real-time sensor data available for educational purposes (e.g., [4,5,6,7]). Despite this availability, however educational success is not guaranteed; non-technical audiences are still likely to struggle when connecting these data streams to physical contexts and phenomena. Research on place-based learning has recently capitalized on the increased availability and affordability of mobile, interactive technologies to enhance learners’ interest and engagement [8]. Many of these applications focus on situated learning in environmental science education [9], and a few others on non-STEM informal learning (e.g., social studies, art, etc., e.g., [10]). However multimodality is rarely incorporated in these applications [11], thus limiting the potential for diverse modes of learning [12]. There is a gap in knowledge regarding the extent to which Cross-reality (XR) and multimodal interfaces in situated STEM learning broadens participation and enables sensemaking from sensor data, to promote conceptual changes and support decision-making. This paper proposes the design and implementation of a system architecture that integrates XR and multimodal data representations to allow different levels and modes of situated learning, that use smart buildings to promote data literacy, computational, engineering and system-thinking skills development. Specifically, we present a learning environment which imparts an increased level of situatedness that allows learners to access, interact with, and experiment with, the environmental and physical data collected by the sensor network, all in addition to interfacing with the building itself.

The central assumption of our design is that situated learning environments in buildings enhance engagement in STEM inquiry and foster public participation in our built environment. This is achieved through the integration of the experience of the physical dimension (i.e., “what”: a building) with a cognizance of underlying technological and scientific dimensions (i.e., “why” and “how”: AEC principles and methodologies), all mediated by immersive technology and multimodal representations. Like exhibits in public settings, the proposed user experience creates conditions for STEM learning activities that otherwise would not normally occur as people use the space as intended. Therefore, the designed learning environment for this building is a useful analog for how a variety of public spaces might leverage inexpensive sensor technology and ubiquitous computing to support a variety of informal STEM learning outcomes for a wide range of audiences.

2 Theoretical justification for using enhanced digital reality to aid learning

While the appeal of Augmented/Virtual Reality (AR/VR) solutions appear to be attractive from a technological standpoint, is there legitimate justification to implement such activities within educational settings? Admittedly, rigorous applications of such technology to STEM learning are still rare, and most existing efforts typically involve direct bestowment of procedural knowledge such as in surgery training or flight simulation [13]. That said, much basic research in the areas of learning and cognitive science supports the notion that broad implementation of AR/VR into educational settings is a worthwhile endeavor. Specifically, AR/VR affords several opportunities to leverage context, visualization, and active engagement to potentially support understanding and learner engagement that is very difficult to attain with traditional instruction methods.

Seminal work on situated cognition (e.g., [14]) has suggested that the context in which learning occurs is critical for accumulating knowledge, and that it is very useful to not only include authentic (i.e., real-world) activities in the learning process, but also a structured means of exploring this information to realize mastery. Much research has demonstrated that context is a powerful memory cue (see [15]), and can influence what we pay attention to [16], how we make inferences [17], and even how we organize and structure what we do remember [18]. AR/VR activities in particular provide learners the opportunity to be physically transported and immersed into such authentic situations without the concerns of cost or safety that are especially prevalent in high-risk job duties. Hardware, environmental factors, and other relevant data can all be effectively simulated and delivered to the learner in high fidelity, at an increasingly lower-cost, with high quality commercial hardware available only recently accessible even to everyday consumers. This provides learners the unique opportunity to interactively engage with these elements in appropriate ways that are not only educationally valid, but also more consistent with their eventual job duties. Such interaction has been shown to produce optimal levels of learning and information retention beyond that provided by mere exposure to information [19, 20], and thus maximizes the subsequent educational impact of coursework and activities. In short, AR/VR provides access to valid learning experiences that would not be feasible or realistic in other traditional educational settings.

Further, by their nature, AR/VR solutions can also incorporate advanced visualizations into the viewed environment that can be used to enhance and augment learner understanding. Doing so provides another range of unique benefits which have explored in prior research in the domain of computer graphics and data visualization. Simply presenting propositional information in a visual form can increase understanding of relationships and connections between concepts that might not be transparent otherwise [21,22,23], and allowing users to also interact with and generate such dynamic visualizations of their own on-the-fly, is also likely to improve understanding of the material [24]. Appropriate dynamic visualizations can also offset individual differences in cognitive aptitude, effectively levelling the playing field to allow all learners, regardless of their background and raw cognitive ability, to achieve higher levels of understanding in STEM areas [25, 26]. As AR/VR activities have the capability to seamlessly integrate data overlays and visualizations into the perceived visual experience in ways that are often interactive and can be tailored to perceive user demands this should enable a more robust knowledgebase that reflects a more nuanced and deep understanding of the to-be-learned situation or material. Lastly, the potential motivational aspects of learning via AR/VR are very significant. It has long been recognized that learner interest is positively correlated with knowledge retention rates [27]. It is distinctly possible that the novelty of learning through cutting-edge immersive technology can be reliably leveraged to foster and cultivate such interest [28, 29]. This is perhaps analogous with recent interest in educational interventions like “gamification,” which have sought to do this exact thing; to teach learners using technology that they perceive as ‘fun’ [30, 31], and thereby create a window to implicitly increase their engagement with and understanding of conceptual material. AR/VR based instruction has the potential to realize this same benefit, which will increase the likelihood they will not only learn immediately, but also continue to accumulate skills for learning faster and more effectively which will be useful for their entire lives in a variety of ways [32, 33].

3 Research objectives

The designed data navigation platform has various goals, hereafter summarized in two main objectives:

  1. 1.

    Present the performance metrics of the building to AEC experts in order to help them make informed decisions for maintenance and repair work in the building and aid their design of future constructions.

  2. 2.

    Enhance data literacy and other STEM skills of laypeople by facilitating access to the sensor-informed BIM and linking it to situated experiences of the physical phenomena (Table 1).

    Table 1 STEM skill areas supported by the X-reality cyberlearning platform and related activities
    Full size table

Examples of experts’ analysis and decision-making specific objectives (main objective 1) enabled or otherwise facilitated by the platform’s navigation tools include:

  • To analyze the effects of environmental loads on the structural systems during construction

  • To provide guidance for moisture management during construction.

  • To detect the effects of any leakage and flaws in the building enclosure during the building’s service life, thereby providing guidance for maintenance, repair, and/or future design.

  • To evaluate deviations from the designed structural form caused by structural loads during the service life of the building, thereby providing guidance for any needed interventions and/or inform future design.

  • To identify damage after exceptional events (e.g., strong winds, earthquake)

  • To provide information on the safety of the building and the need of any repair.

Examples of STEM content delivered through the cyberlearning platform (main objective 2) include:

  • Data content:

    • Analysis of parameters describing heat transfer and storage in materials and building assemblies (i.e., heat flux, R-values);

    • Analysis of parameters describing moisture transfer and storage in materials and building assemblies (i.e., wood moisture content distribution);

    • Analysis of deflection and strain (normalized deflection);

    • Descriptive statistics for measuring physical characteristics (averages, trends, etc.);

    • Analysis of measurement precision (i.e., analysis of outliers, erroneous readings, etc.).

  • System concepts:

    • Conceptual relationships between building dead load and deflection in structural members;

    • Relationship between outside and inside temperature and temperature gradient in between;

    • Analysis of moisture gradient over time as a function of external and internal conditions.

  • Process concepts:

    • Analysis of dimensional changes due to moisture absorption/desorption (swelling/shrinkage);

    • Analysis of internal stresses due to temperature and moisture changes.

To achieve the main objectives, three main tasks have been defined as critical for the design:

  • Identification The search for the occurrence of all specific values, trends, anomalies, and extremum for attributes in specific locations at specific points of time.

  • Comparison The comparison of the measured values for all attributes in spatial and/or temporal intervals.

  • Correlation The characterization of changes of all attributes by observing their distributions over spatial and temporal spaces or subspaces, especially in relation to other attributes.

The following requirements have been considered for the proposed platform:

  • The raw data must be presented in an intuitive and accessible manner;

  • The user should be able to interact meaningfully with the BIM without extensive training;

  • It should be possible to navigate in both spatial and temporal senses to explore the measurements at different locations in the building and at different points of time;

  • Through the audio and visual modalities, the user should be able to detect and extract underlying patterns, trends, and divergences from the expected behavior.

  • The user must be able to detect the semantic relationship between the sonification and visualization of the data.

4 Review of literature

Public access to structural health monitoring (SHM) data and models is imperative to STEM education and scholarly engagement by improving trust in science and engineering among laypeople and allowing them to participate in decisions about the management of built environments [34, 35]. To provide such access, advance information visualization technology has recently been employed thanks to its responsiveness and capability to easily and effectively communicate complex concepts [36]. As such, development of (and research into) this type of innovation will enable optimal experiences for lay people in the built environment [37]. One form that this new technology has taken is known as “Cross Reality” (XR), which encompasses virtual reality (VR), augmented reality (AR), and projected displays (“spatial augmented reality”—SAR), which has enabled an entirely new way of data sensemaking [38]. However, innovations which employ this technology are only budding and subject to severe limitations [39,40,41,42].

Concurrently and independently from these advances, real-time digitization has only just barely begun revolutionizing AEC [43, 44]. Past limitations to this have mainly related to the lack of available digitization of construction sites and buildings with sensor networks. Outside of AEC, real-time presentation has been shown to considerably augment the spatial perception of abstract data in XR. However, despite the potential support this type of perception provides to STEM education, the intersection of AEC and these postulations is as of yet unexplored [45, 46]. Succinctly, there is a gap in the knowledge required to engineer XR systems and SHM data platforms that are effective at facilitating education among non-professionals.

4.1 Representing data through sound

Sonification has been defined by Kramer [47] and colleagues as “the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation”. However, the use of sound to represent data is not nearly as widespread as the use of graphics. Arguably, graphics are better equipped for processing and providing an overview of large amounts of data at once. On the other hand, sound is perceived sequentially, and making sense of nuances in these sequences is difficult, especially if they are unfamiliar [48]. Also, in contrast to the field of information visualization where design principles have been established and are in constant review [49], there are few agreed-upon principles related to the design of sonifications [47, 50, 51]. However, sonification has significant untapped potential. Human hearing system is capable of discriminating miniscule changes in pitch and tempo, which can be used to represent data in a easily perceptible manner [52]. Furthermore, auditory representations perform exceptionally well regarding trend analysis, the estimation of data values or comparison, and pattern detection [53], and can highlight subtle or gradual value changes and emphasize anomalies and outliers [54]. Nesbitt [55] suggests the representation of temporal data be one of the most promising use cases for sonification (e.g., rhythm can represent changes of data across time). Nevertheless, while previous research has highlighted the potential of sonification, little is known of how people experience, characterize, and interpret sonifications at a cognitive, somatic, and emotional level.

Responding to this gap in understanding, our research investigates how people characterize their experiences and interpretations of sonifications coupled with visual representations.

4.2 Representing in-situ data through visualization

Because of increasing interest in in situ interaction, data visualization became one of the key components of interactive AR systems [56]. The visualization of such data, however, has not been comprehensively investigated in the AR research community. Most AR approaches attempt to solve only problems caused by data overlaying and cluttering in the real scene [57,58,59,60,61], whose implications not only affect information understanding, but also perception [62,63,64]. However, this method presents a cumbersome disconnect between context-specific information and overall information. The understanding of information from sensor data is also affected by the “clutter challenge”, as the background layers interfere with users’ perceptions and might make specific data elements within the layer difficult to perceive. As such, selecting a location and data representation that does not conflict with the visual background and also retains the relationship between the virtual and the real scene is a central challenge in the development of AR platforms.

In the fields of AEC, there is some initial work which might support such efforts. For example, [65] developed a wearable device called “iHelmet”, consisting of a mini projector and an iPod touch mounted on a security helmet, to project building-related in situ information on walls of construction sites. Although they found that this device improved the presentation capacities, portability, legibility and access to the presented information, the limited interaction possibilities restricted its usability significantly. Riexinger et al. [66] more recently introduced a system that provided instructions and monitoring information during planning and production/construction processes. The interactive in situ information was presented in XR on multiple platforms. In-situ visualization has also been investigated in terms of its possible value for educational purposes. Kotranza et al. [67] developed and tested an in situ AR visualization system for clinical medicine training. Sensors were used to measure learners’ performance and present guidance in real-time. Supporting information is then presented in form of in situ visual feedback. The ongoing research of Messadi et al. [68, 69] investigates how MR based visualization of in situ data affects the motivation of students and their progress towards learning outcomes in a collaborative, interdisciplinary learning environment. Their preliminary results showed promising improvements in the performance of students assisted by the visualization compared to the control group.

Coupled with the sonification discussion above, in this manuscript we demonstrate that the use of multimodal presentations (e.g., sonification and visualization) can streamline the learning process related to SHM comprehension. The combination of visual and auditory modalities can provide perspectives that foster unique insights into data by overlaying a multimodal data representation (3D data of the inspected building component, local sensor and related data, and associated sound progressions) and situating data in the physical space, thereby supporting learners in their efforts to construct meaning.

5 Case study

5.1 The Peavy Hall environment

Innovation in the construction sector is often driven by pioneering buildings, where very sophisticated performance criteria are applied, and performance data is evaluated in great detail. Exemplifying this is the George W. Peavy Forest Science Center (“Peavy Hall”), which is a new location of the Oregon State University’s College of Forestry. It is an example of pioneering timber building which will serve as a full-scale laboratory (“Living Lab”). It will enable field testing, monitoring and sharing of data on the in-service performance of engineered wood products and advanced, earthquake-resistant timber construction systems. While the original purpose of the instrumentation was to provide data to experts and interested investors [70], the Living Lab also presents a unique opportunity to create a highly specialized and, publicly accessible learning environment.

In order to develop cost-effective and reliable XR applications, it is imperative for designers to cross-check assumptions made during the design phase with the conditions that the structure will actually experience. This is especially important when a structural system is at its early stage of adoption. Outside of the US, research and educational institutes, such as the École Supérieure du Bois (ESB) in Nantes (F) [71], the ETH in Switzerland [72], the UNBC [73], and the UBC [74] in Canada, have initiated programs to promote the increased usage of cutting-edge, sustainable timber structures and technologies. These initiatives include the construction of new facilities made of engineered-wood products and the development of research programs for monitoring the structural performance and hygrothermal conditions thereof. Just one of the above-mentioned projects has foreseen public access to data and implemented an AR navigation platform for building users [75].

The “Living lab @ Peavy Hall” employs a network of sensors through which quantitative data on material and structural parameters is continuously collected. While the Peavy Hall is under construction at the time of writing, our cyberlearning design has already shown utility for both educational purposes and to help “tell the story” of the building [76, 77]. The ability to coordinate design and installation of the monitoring infrastructure during construction of Peavy Hall in collaboration with the building design team, contractor, and industry leaders in advanced sensing technology, has helped greatly to appease stakeholders’ requirements (e.g., knowledge needs). Additionally, the installation of some sensors during construction allowed us to capture and communicate data on some dynamic phenomena evolving during the construction phases, as opposed to steady-state phenomena in service. This has also greatly facilitated our ability to conduct preliminary development activities by providing us with real-time in situ data to experiment with.

We designed and implemented XR infrastructure to support learning related to the construction phases:

  1. 1.

    To analyze the sorption/desorption behavior (MC-moisture content data) of Cross-Laminated Timber (CLT) panels during different construction phases and to relate this information to:

    1. a.

      meso-climate conditions on site (weather station data)

    2. b.

      micro-climate conditions in the proximity of the measuring points (i.e., spatial data: orientation towards prevailing winds, cardinal orientation, shading and sheltering effect of other buildings/building components)

  2. 2.

    To evaluate the evolution of some structural parameters (e.g., loads applied on post-tensioned steel rods, and following strain relaxation)

5.2 Interaction design

The aim of the interaction mechanisms is not simply to offer a way of accessing the application and assessing the data. It is also a form of support for the sense-making process. The more ways a user can explore, transform, and experiment with the data, the more unique insights are able to be obtained [78]. In the case of an XR application, where the user immerses into environmental data rather than only observing it, the interaction methodologies are critical to the quality of user experiences.

In the proposed learning environment, the user interactions serve several main purposes. Firstly, they allow the user to intuitively “wander” the virtual environment in spatial and temporal senses. Furthermore, they permit the user to browse the data, switch between the data representation modes, modify the system parameters, add or remove virtual components, select and view rendered objects, move or manipulate scene elements, etc. Lastly, the interaction techniques which go through the virtual system can have an impact on the real physical state of the built environment, allowing users to open windows and doors, adjust lighting, and digging into data.

The interaction possibilities depend strongly on the hardware capabilities of the utilized platform. Therefore, for each platform, different types of interactions have been defined. In the AR and HMD-VR modes, for instance, the application reacts to the input from the built-in sensors of the HMD or mobile device and allows the user to intuitively “look around” in the scene. In the desktop-VR mode, on the other hand, mouse and keyboard, 3D controllers, or multi-touch displays are used for this purpose. Similarly, depending on the platform, the interaction with menus and other interactive elements of the scene are performed using a mouse, touch functionality, VR controllers, the viewfinder camera, or hand gestures.

5.3 The design of the learning environment

Three immersive modalities have been designed and implemented in Peavy Hall, aiming to increase the physical significance of educational experiences (see Fig. 1):

Fig. 1
figure 1

Cyberlearning environments organized for levels of situatedness. a Off-Site Helmet Mounted Displays (HMDs), b A Mobile Device as a Viewfinder, c Data Projection into Physical Space

Full size image

5.3.1 Off-site helmet mounted displays (HMDs)

Using a commercial VR-HMD in any location, users can interact with the bountiful data set associated with Peavy Hall in a fully virtual reproduction using our application. This is achieved through creating a digital twin of Peavy Hall [79] in real-time using data from the sensor database coupled with architectural CAD information and physical models. Using this, Peavy Hall can be inspected freely with an enhanced degree of user autonomy compared to a navigation of the environment in building itself space thanks to the lifting of physical restrictions which we may provide in our software. Using these abilities, users are free to explore and experiment with the data as they choose. In addition, this allows for unique and creative simulations to be run which could vastly enhance educational capabilities of the system. The research team has already explored the prospects and prototyped a fire-disaster simulation within Peavy Hall, for instance.

5.3.2 A mobile device as a viewfinder

By using a handheld viewfinder, like a mobile phone, tablet, or convertible detachable laptop, the user can be immersed in an AR version of the physical world. As the device’s camera is pointed at elements from physical space in Peavy Hall, sensor and building information are displayed into the scene accordingly. As much computation as possible is handled by a server so as to minimize the work necessary for the user’s personal device.

5.3.3 Data projection into physical space

Through projection mapping, a visualization of the sensor network’s recordings can be made into a tangible and public experience. That is, a photo projector will cast relevant visualizations of sensor data onto the physical surfaces which the data relates to. A projection mapping of the rendering will be performed under the appropriate dimensions and constraints of the geometry being cast upon. Furthermore, users can modify visualization parameters in real-time with their personal devices or terminals set up in public spaces. These parameter modifications will be displayed in real-time to the others in the area, allowing for “shared” experiences. Dynamic access control ensures that each single object is only manipulated by one user at a time.

In all the aforementioned platforms, we use sonification techniques [80] to map sensor data parameters and create an auditory rendering of data. These are grouped into three classes:

  1. 1.

    Iconic sonification Mapping data to sound effects, associated with certain user driven event.

  2. 2.

    Audification Playing data directly as a binary sound file.

  3. 3.

    Musical sonification Mapping data to musical parameters, such as pitch register, tempo, and volume of individual instruments. In addition, different music instruments such as harpsichord, shamisen, and a synthesized pad will be combined to explore multidimensional data properties. The coexistence of multiple tracks, which is one of the features of digital music, will be used to produce spatial distributions in temporal angles.

Both visualization and sonification are interactive interfaces, which allow users to control data to better understand the information being presented [81].

5.4 System specification

The software facilitates data representation in XR by offering the following main features: (1) Navigation in and around the virtual representation of the building, (2) data visualization at the spatial location of data acquisition, and (3) intuitive user interaction with both the physical and virtual spaces. The virtual representation of the building is generated by importing 3D models created in Google Sketchup and other CAD software (Rhino and Revit). Combining the geometry of the 3D model with the data allows us to link the data to its spatial acquisition location in order provide the user with precise in situ visualizations in each platform. The data is consolidated into a virtual BIM from an array of sensors installed in Peavy Hall that measure various physical properties. The user can opt to visualize the data collected by a specific sensor by selecting the sensor within the scene with the device-appropriate interaction mechanism. The data is loaded to the system from a database in real time, which allows the user to view a digital twin of the building and monitor its state off-site. Furthermore, the user has access to historical data and can observe its patterns and changes over time.

5.5 The design of the building instrumentation architecture

A multi-purpose sensor system has been deployed at strategic points throughout the building: Three levels of the building and the roof, interior and exterior (see Fig. 2).

Fig. 2
figure 2

Localization of the selected monitored areas

Full size image

The building instrumentation includes:

  • A weather station (currently on the construction site, but it will be permanently set up on the top of the roof) to collect data on outdoor climatic conditions both during construction and in service.

  • Ambient Temperature and Relative Humidity (RH) sensors to monitor indoor climatic conditions.

  • Heat flux meters, installed in selected external wall and roof sections of the building, to keep track of the heat and moisture transfer.

  • Wood moisture meters and surface temperature sensors, embedded in selected locations in engineered wood and frame members.

  • Moisture sensors will reach different depths in the members, to monitor possible moisture gradients in the layups.

  • Displacement gauges to track vertical and slip movements due to shrinkage of floor panels and deflection (creep).

  • Strain gauges and load cells to measure the post-tensioning losses in the shear walls.

  • Temperature sensors in the shear walls to monitor local/surface temperature (Since the wall compressive deformation measurements and the thermal extension of the post tensioning steel are temperature sensitive).

  • String potentiometers, installed in selected joints in the shear walls and between walls and floors, to monitor joint openings and floor deflections.

  • Three-axial accelerometers, located on each floor (plus one on the foundation) to identify building accelerations and characterize global dynamic behavior (seismic monitoring) and local floor vibrational performance.

5.6 The Peavy Hall system architecture

The cross-reality environments uses existing tools and standards available for sensor and model data exchange (e.g., ISO 16739:2013; ISO 29481). Sensor data has been exposed through the development of a middleware capable of ingesting and interpreting data sets into interoperable communication standards such as WMS, WFS, and WPS. One of the goals of the implementation plan was to identify, collect and harmonize all information inputs already available to support the project’s needs, including multimedia- GIS- CAD-BIM, and geo-located and time-stamped monitoring data. The software architecture, as illustrated in Fig. 3 [79], has been designed according to a modular methodology based on a federated approach to data collection, processing, and distribution of spatial–temporal data. Following this tiered approach, we have designed and implemented: (1) an application layer, (2) a middleware layer, and (3) a data layer. Our challenge has been to create a system capable of sharing digital learning experiences and self-generated content among users and of improving the experience of retrieving and presenting information from SHM data. For the development of the project’s technical platform, we have followed an iterative procedure. Focus groups have been established at the early stages of the project to address the needs of various project audiences (i.e. professional, public, special needs).

Fig. 3
figure 3

Pictorial representation of the 3-layer architecture of the proposed system

Full size image

User requirements are iteratively analyzed in terms of usability, data accessibility, and experience-level/scaffolding by continuous interacting with the researchers and AEC professionals involved in the building project. To create the XR platform, we have adopted two real-time 3D rendering engines: Unity and Unreal Engine. The real-time nature of the user experience is imperative, as a user must be able to move through a physical (and virtual) location, have relevant information drawn from the local sensor networks, and then have BIM metrics presented in the most medium-appropriate format. Additionally, the following commercial software has been extensively used during the project: Revit and Autodesk Navisworks (to represent the various sensors into the Revit model), and SurfaceMapperGUI combined with Processing (to render visualizations of sensor data).

5.7 The design of the visual experience

The design of the visual experience of the XR application plays an essential role in achieving its objectives. Several requirements that have been addressed in this process are described in the following.

An appealing and exciting visual experience not only can encourage people to use a visualization system, but it may also keep them engaged and interested while using it. Moreover, the aesthetic appeal of any kind of system is key to enhancing usability and improving user experience. Studies have concluded that a product’s perceived aesthetic quality highly correlates with its perceived usability [82,83,84]. It has furthermore been shown that products’ visual appeal positively impacts user’s performance when using it [83] and their perception of its trustworthiness [85]. Therefore, the visual appeal of the system was of utmost importance in our design.

Imparting data literacy is a major objective of our system. Accordingly, the designed cyberlearning environment aims to train people to understand both scientific and non-scientific data. This requires legibility and intelligibility of all the data representations which will be presented to the system’s users. Similarly, due to the high volume of associated with the BIM, our system requires mechanisms for averting clutter, information overload, and misperception.

Creating an environment conducive to informal learning is a further central goal of our system. It is known that dynamic visualizations outperform conventional, static instructional materials for informational and educational purposes [86]. To take advantage of this, our proposed system offers interactive, animated visualizations in an immersive environment. Such an environment has a multitude of advantages over those generated by conventional learning materials. First, technology-mediated learning approaches appeal to many learners, improve their attitude and satisfaction, and have a motivational effect [87,88,89]. Second, dynamic, digital materials can contain considerably more information than print materials, in addition to being able to augment cognition by supporting multimedia, immersive, and 3D data representations [67, 90, 91]. Lastly, freedom from spatial and temporal restrictions provides many opportunities that are physically infeasible in the real environment: The learners can be present at locations that are otherwise physically or geographically unreachable, examine complex and intricate graphics in an extremely short time, and be provided features such as:

  • A bird’s eye view.

  • Navigation through time, enabling the learner to view both historical and predicted future data.

  • Scaling up and down the visualization allows for obtaining a proper view of the data.

  • Speeding up and slowing down the time expiration can reveal long-term patterns and short-lasting nuances.

The performance and scalability challenges of our system must also be addressed. Rendering high quality 3D graphics requires copious amounts of primary memory and CPU/GPU resources. This can become a bottleneck of the system, especially in case of mobile devices and other equipment with limited computational resources. Since the system visualizes the real-time data dynamically, the performance requirements become even more severe. Operating on the high volume of the historical data, collected by several sensors over longer periods of time, creates a challenge of a similar nature. Hence, scalability of the system is another consideration in our design. Possible solutions to issues which we are considering include: Simplifying the geometry of the objects as much as possible without compromising the rendering results; limiting the visualizations to the visible areas of the scene; loading small portions of the data based on demand to prevent unnecessary processing and blockage of the system for loading larger sets of data.

In the current version, the user can view a 3D model of the built environment, navigate in and around the building, and choose between different modes to view the moisture content, displacement, and tension loss of CLT walls. Our abstract modes of data visualization take on the following three forms: texture-based, particle-based, geometry-based visualization. A description of these modes is given in the following.

5.7.1 Texture-based visualization

In the texture-based visualization mode, sensor information is represented as a heatmap which is applied as a texture to the surface of the digital-twin entity the data pertains to (see Fig. 4). The color is set with respect to the data value at various corresponding locations on the objects, and then computed across all points of a grid spanning over the surface. The specific model by which this computation is performed varies both based on the type of data, physical properties of the object, and simulation parameters defined by the user. This type of visualization is used so far for:

Fig. 4
figure 4

Examples of representing the moisture content of a CLT wall with a heatmap in both the virtual reality and augmented reality setups. The locations of sensors are indicated with cyan-colored virtual objects rendered in each respective environment

Full size image
  1. A.

    Moisture content data at different locations in a single object.

  2. B.

    Displacement values recorded by the string potentiometers (interpolating between displacement values at fixed points and values at the measured locations).

  3. C.

    Tension loss data recorded at the tendons in the shear walls. For this purpose, the measurements from multiple sensors are interpolated over the measured object.

These values are recomputed for every new sensor database query. The sequence of measurements is presented to the user as an animation to visualize the temporal variation of data. To ensure a smooth distribution over the set of possible colors, a feasible range of values is defined specific to the data type.

5.7.2 Particle-based visualization

The particle-based visualization represents data by distributing varied spheres over the object at fixed grid points. To make these “particles” visible inside the object, its texture is made transparent. Using the same mechanism as the texture-based mode, the values are calculated at all grid points based on the sensed values and are recomputed for each new measurement. At this stage, two different particle-based graphics have been implemented.

In the first implementation, the value at the grid point determines the color of the corresponding particle via the same algorithm employed by the heatmap mode. Since the particles are viewed on a 3D grid, this visualization provides a more detailed overview of the distribution of values compared to the texture-based visualization. This is particularly useful for data with three-dimensional spatial significance (e.g., moisture content values at specific locations in the plane of a panel in addition to various depths.)

In the second implementation, as shown in Fig. 5, the data informs the size of the particles instead of their color. The scaling is set with respect to the predefined range of feasible data, which can result in slightly overlapping particles at areas with highest values. Accordingly, these areas seem opaque to the user, while low-value areas appear transparent. The rationale for offering this mode is that, depending on the type of the visualized data, representing values with a size variable may be more intuitive to the user. For instance, it may be easier for the user to learn about moisture through size rather than color, while in contrast, visualizing temperature with a colormap may be more familiar than size due to the omnipresent use of this representation.

Fig. 5
figure 5

Example of representing data with a particle system in both the virtual reality (a) and augmented reality setups (b). The size of the particles is proportional to the moisture content of the CLT panel

Full size image

5.7.3 Geometry-based visualization

In this method, the data is used to create or modify a geometry (represented by a mesh in the 3D model). For example, the moisture content of a CLT wall can be represented as tubes passing through the wall. The radii of the sections of the tube is proportional to the moisture content at different layers of the CLT (see Fig. 6). This visualization is designed to represent the measurements at a few locations of interest, rather than over the whole object. This can directly draw the attention of the user to notable locations, such as the actual location of the sensors, in a way the other modes cannot.

Fig. 6
figure 6

Examples of representing data in the augmented reality setup by procedurally generating a geometry that corresponds to the sensed values. In this case, the created geometry is a tube that has varying radius at different sections. These radii correspond to the values measured by the moisture sensors located on the same object or in the same location

Full size image

A further possibility for geometric visualization is the direct manipulation of the geometry itself with respect to the measured values, which can provide a unique spatial insights regarding the data. For example, to visualize high moisture values on a CLT wall, the respective areas of the wall can be “swollen” by manipulating the wall’s mesh accordingly.

5.7.4 Spraying the visualization

After synthesizing the data visualization, it can be presented to the user in two different ways. The most straightforward way is to make it immediately visible to the user, such as by replacing the underlying texture of the object with a heatmap-texture. The second way is to allow the user to choose the regions of the object where the visualization should be visible. The method is similar to spraying paint on the object and is described in the following.

This visualization approach enables the user to limit the visualization of the data only to specific regions of an object that are of interest to them. An example of the application of this mode is depicted in Fig. 7. Initially, the scene is rendered with the underlying textures, appearing just as the physical building. With the help of the supported interaction devices (e.g. mouse or VR controller), the user can mark areas of the scene where they would like to see the data, which modifies the transparency value of the textures. Consequently, the visualization blends with the underlying texture according to the transparency values. To the user, this process appears as spraying paint on the object except with the painted color depends on the sensed data instead of simply being a static color.

Fig. 7
figure 7

Example of spraying the visualization. The visualization is only visible at areas that are of interest as selected by the user using a virtual paint spray

Full size image

This approach has various advantages over the default visualization. First, it gives the user the ability to constrain the visualization to the region of interest and blend out the other parts. This reduces the visual cluttering of the scene and aids perception. Furthermore, by giving the user more autonomy, the immersivity of the application is deepened, which supports the engagement of the user and is conducive to a more appealing experience. Moreover, by actively involving the user in the process of displaying the data, information retention is enhanced due to the heightened attention overhead.

5.7.5 Simulating the visual effects of physical phenomena

We aim to provide intuitive and easy-to-interpret data representations that don’t require extensive user training to perceive or understand. One way to do this is to imitate the impact of different physical events and environmental properties on the material with visual effects that are commonly understood. For example, moisture on a wooden wall can be visualized by overlaying a water texture (Reflective, darkened, beady, etc.) upon the base timber one.

Such visual effects can also change, intensify, or fade over time to simulate the temporal states and changes of the object given relevant historical data. Furthermore, by using proper simulation models, the effects can be extrapolated to the future, which allows for the visualization of predictive models. This can support education of a wide variety of AEC concepts.

5.8 The design of the auditory experience

In tandem with the visualization, a sonification feature aims to further improve the user’s experience and education in data literacy. Sonification of all sensors placed in Peavy Hall is available in either “passive” or “active” user mode. In the passive mode, the user is unable to change the aural representation while experiencing it. Data sets (moisture content at various sensor locations, date, temperature, etc.) are shaped, modified, and mapped in advance of the user experiencing the MR environment. In the active mode, the user can change the aural experience in real-time. The central sonification interface, built in Max/MSP, receives data from each sensor, reads it in real-time, maps it to sound parameters, manipulates it to best suit each sound parameter, and interfaces with the VR engines and user devices to provide interactive operator control. Central to the sonification interface is the method in which data is stored, read, and presented: A sequential counter component holds the sensor data from all thirty-eight data sets and reads each at a user-defined rate. How data is parsed, manipulated, and mapped is unique to each sonification mode and will be discussed in more detail below.

5.8.1 Sonification modes

The sonification exists in three forms: an iconic environmental sonification, a musification sequence, and an audification soundscape. We chose to design an environmental sound score as a backdrop to the VR experience so as to tailor the aural surroundings unique to specific locations the user experiences within the visualization. The current shaped sound design emulates natural environmental sounds native to Peavy Hall’s geographic location: Corvallis, Oregon, a small residential city in the Pacific Northwest. Other aural backgrounds may be substituted in as well, including a forest, sea, cave, and desert, as well as non-standard user-designed ambiences. This can be used for the design of specialized sub-experiences within the XR environment. Static iconic sound effects with clear connotations to location and weather conditions were selected because it is our hypothesis that including and manipulating these familiar sounds assists rapid and accurate data perception for a wide range of potential users. Our evaluation plan will include an analysis of the effectiveness of this variance in ambience.

The sound score for our environmental sonification features samples of real-world sounds, including wind, rain, light traffic noise, insects, and birds. Date, time, and temperature data influence gradual environmental changes during the eleven-month data collection period. Common audio transitions include the sound of heavy wind and rain (as heard during the months of April and June) yielding to birds and light traffic noise (as heard during a sunny day in July). The sensory data measuring moisture content values at various positions in the building is modified and mapped to sound parameters, which include amplitude, pitch/rate, depth, and width (via reverberation control and panning, in tandem with amplitude control) of each individual sound file. After an additional round of modification, all data is scaled to best fit the desirable parameter range before being mapped directly (i.e. most sensor values for moisture content values range from 0.–26., but spatial panning in our application has a range of 0–127). Conditional measures are used to evaluate time and temperature information in order to trigger crossfades between sound files that represent environmental soundscape changes to match the relevant weather conditions.

The second form of sonification used in this project is a musification derived from selected sensory data. In this example, a pre-existing, eight-bar musical pattern is generated and repeated continuously by three instruments: Harpsichord, shamisen, and a synthesized pad. Moisture content and temperature data values are mapped to influence changes in the tempo of the sequence, as well as to the volume and octave register of each individual instrumental part. For example, when the weather is warm and dry, the musical patterns play in higher octaves. Future implementations will use date, time and temperature data to alter the harmonies of the musical sequence (i.e. minor mode when it is colder/raining, and major modes during warmer weather). The sound sources used in our current version of the application are played in General MIDI (GM), which is ideal to free up resources for the heavier audio processing requirements of the other two sonification modes. The musical progression is hypnotically minimalistic so as to allow the user to focus on the change of a few sonified parameters while allowing the dominating aural elements of the music to remain intact (melody, rhythm, and duration). This successfully supports the discovery of pattern within the data through musical sonification [92].

The last mode in the sonification platform is audification. It takes data from all thirty-eight sensor sets and directly applies them to sound parameters with some filtration. An audio synthesizer engine was built featuring additive synthesis and frequency modulation synthesis (FM) to generate sound comprised of electronically-generated waveforms and filters that are combined, multiplied, and fed back into each other to create an intricate sound mass sequence. Each value from each data set is triggered in a linear sequence (with a variable read time) and mapped to sound parameters. Some examples of sonified parameters are as follows. Temperature data from various sensor is mapped to influence the frequency of carrier, multiplier, and sub-audio oscillators. Moisture content from the upper portion of the rooms is mapped the cut-off frequency and width of a state-variable audio filter, while lower sensors are influencing the amplitude of each of the four generators used in the synthesizer engine.

5.8.2 User interaction

When the sonification interface is in active mode, the user can influence how data is processed in real-time. As in the visual interface, the user can target a sensor, physical element, or group of sensors for further data investigation and presentation styles. Depending on enabling technology, this could be done by moving closer to a sensor(s), clicking with a tablet, or selecting with a mouse. The sonification can also be automatically linked to the visualization system (selecting a specific sensor in the visualization has an automatic built-in parallel mapping in the sonification interface), explored independently, or custom mapped by the user.

The interactive element of the user control is an intuitive means of facilitating greater data perception and retention for both experts and laypeople alike with minimal training thereby enabling a unique sensational modality to experience data with. The user selects specific sensory data and can choose how to apply it to any or all of the available sound parameters. In active mode, user defined sensory mapping is available in all three sonification modes in real-time (sonification, audification, and musification). Passive mode, on the other hand, allows users to experience for certain predefined experiences or lessons. All modes are available for both individual and group listening. In order to minimize clashes between multiple audio layers, a “focus mode” allows users to select a few parameters and minimize the volume of the other elements. An example of this could entail isolating the sonification of moisture content of the upper beam sensor to influence the respective octave register an instrument is playing in the musicification. This instrument would receive greater amplitude presence (among the other instruments, as well as musification receiving more presence than the other two modes), while the other sounds are quiet and suppressed in the background.

Finally, the method and rate by which data representations are played back is user-alterable. Users may define the intervals and time between intervals in ranges including milliseconds, hours, and days. Hence, the user can experience months of data in minutes or seconds. The user may also choose specific ranges in each data set or select from a few presets (winter, rainy day in July, December, etc.). This can be used to provide unique perspectives on data trends which aren’t obvious across very narrow ranges of time.

We believe that our approaches to data literacy enhance learning with the addition of sonification. All of the sonification modes selected rely on familiarity with sound material and trivial training. The environmental sonification places the user in familiar territories with iconic connotative sound sources, while the musification and audification modes features extensive repetitions so that the user can learn patterns and attune themselves to their changes. Focus mode allows the user to isolate specific parameters for closer study. Working with the visualization control, we are motivated by many similar potential discoveries to which this XR platform can lead.

Figure 8 shows system architecture of the prototype for the performance and control interface of the sonification element of our project. This is the top-level portion of the application, which is designed for monitoring, influencing, and performing the sonification in real-time. Each of the areas coded in peach is a module that provides useful front-end features for the several sub-patches in this project. The objects with purple borders contain several other functionalities. In its current state, the interface is broken into 4 modules: musification, audification, environmental sonification, and the data set control component.

Fig. 8
figure 8

Peavy Hall Sonification Interface

Full size image

6 Conclusion

The intersection of AEC has hardly been supported by the technologies presented by what is known as the “4th Industrial Revolution”. However, due to increased demand for building sustainability and responsiveness to occupant needs, sensor networks to collect performance data of buildings and more comprehensive building models have recently been developed to represent geometric and semantic information within simulation, analysis, and optimization processes. Because of this, it is imperative to train both current and future AEC professionals in the understanding of these data and models. This also provides a unique opportunity to also enable the broader community to involve itself with the management of built environments. We present the design and implementation of a cyberlearning environment where the application of the XR technologies to AEC, supported by appropriate multimodal representation, facilitates data literacy in both professionals and laypeople alike. More specifically this paper details the methodology behind the design of a XR environment in a “smart” building in order to support professional and informal STEM cyberlearning.

A core principle that has driven our design is that hybrid learning environments in buildings enhance engagement in engineering and science inquiry and foster public participation by integrating the experience of the physical dimension with a cognizance of underlying technological and scientific dimensions. While designing such an environment is certainly a challenge, these efforts are mediated and supported by the addition of immersive technology and multimodal representations.

We have validated our central hypotheses by pursuing the following specific aims:

  1. 1.

    To develop scalable and generalizable methodologies for developing XR learning environments that enhance understanding of scientific data related to the built environment through situated learning.

    1. a.

      Our working hypothesis was that the situated, contextual experience of sensed data from buildings enhanced (1) the content, (2) the system, and (3) process thinking and learning in STEM, and that level of situatedness defined in terms of types of experiences with different research platforms (multi-touch, mobile and spatial AR) will have differential impacts for learners.

  2. 2.

    To develop compelling multimodal data communication methods and evaluate their impact on learners’ interest and participation.

    1. a.

      Our working hypothesis is that sonification of data will support and enhance development of (1) content, (2) systems, and (3) process thinking and learning in STEM for all participants.

The pursuit of these aims yielded the following outcomes:

  1. 1.

    Three different mixed-reality platforms for data exploration with different levels of “data immersion” in the built space.

  2. 2.

    A sonification tool designed for the above platforms and tested for different audiences (including visually impaired people).