Keywords

1.1 Introduction

The evolution of Smart Cities may assist facing a wide variety of societal challenges in the coming decades, stemming mainly from over-population, lack of resources, energy overconsumption, waste management, but also critical dimensions such as safety and security. However, there is still a gap between what technology offers and the requirements imposed by such challenges. Building blocks of existing smart city infrastructures are the virtual sensors and actuators, which abstract physical objects and their virtual behaviours on top of their physical networks. The coexistence of, maybe thousands, Physical IoT devices and the corresponding data (distributed in IoT environments) inevitably result into a vast amount of data that needs to be processed and reasoned upon. In such context, Big Data analytics and advanced visualization techniques are a necessity towards dealing with system-level problems that cannot be solved by conventional methods and technologies.

Big Data produced by an IoT infrastructure in a smart city, in its raw form, is difficult to handle, as it rapidly grows very large and complex. Understanding a pattern from incoming data is a challenge. The huge amount of data can be understood only through a series of analytic steps, and visualization holds an important role in this strategy. Bar graphs and pie-charts do play a useful role in understanding Big Data, but they can only provide the headline figures. Cities are already facing difficulties in extracting and visualizing useful information from Big Data to make decisions (Greco and Cresta 2017). This challenge does not concern the technology itself, but the tools and components required to present Big Data in the form of useful information to users, and especially city authorities and decision makers. According to Yafooz et al. (2016), the two main challenges in Big Data visualization are as follows:

  • Human perception: when the number of data entities is very large, users will experience difficulty in extracting meaningful information.

  • Screen space limitation: when large data are displayed on the screen in terms of data items, the screen space will be insufficient.

Recently, Virtual Reality has emerged as a medium to assist with such challenges by enhancing the user experience regarding perception and presence and by immersing the users in a virtual world, where vision and hearing are captivated and elided from the real environment (Tan et al. 2015). Using Virtual Reality techniques, it is possible to build 3D digital models for any real or virtual IoT “object” and then visualize, enhance and share these objects. In this context, the proposed work aims to fill-in the gap of innovative real-time smart city infrastructure visualization and management in the context of Big Data, by combining state-of-the-art 3D visualization techniques and rich interaction.

The work presented in this paper aims to transform traditional smart city data interfaces to easily perceivable rich visualizations, which are characterized by the following:

Immersive data visualization: immersive data visualization is a new trend used by data scientists to perceive and manipulate data in VR, as stereoscopic rendering opens up a world of possibilities. Data visualization tools employing VR displays can be coupled with haptic or kinaesthetic interfaces, making it possible to capture body language, human touch and expression. This allows for intuitive data understanding and straightforward pattern recognition even among enormous data sets.

Increased efficiency through cross-referencing data: VR enables users to stack relevant data, create correlations and visual cues so that they can cross refer it instantly, allowing each user to view huge amounts of data at a glance. The information volume is expected to expand massively and VR can facilitate data perception and analysis through efficient visualizations that take advantage of three dimensions.

Rich interaction: virtual reality allows users to be absorbed in the context of data visualization. Users can manipulate data streams, push windows around, press buttons and actually walk around data worlds in a more direct, immersive experience. In order to enhance immersion, intuitive and rich interaction techniques should be employed so as to facilitate the manipulation of 3D environments, thus allowing users to make more accurate data analysis and ultimately, faster decisions. Interaction facilitates the manipulation, perception and comprehension of data in real time.

In the context of this research work, mid-air gestural interaction was chosen as a natural means of interaction without additional equipment. Apart from being intuitive, gestures were chosen as an interaction technique due to the rich vocabulary they can support, allowing the complex process of manipulating three dimensional environments. The proposed setup includes a Leap Motion sensor placed in front of an Oculus Rift, allowing robust hand and finger tracking through computer vision software.

1.2 Related Work

Smart cities and IoT envision a future, where the sensing and actuation functions blend into the background; and the access to new information sources offers new capabilities (Gubbi et al. 2013). Visual analytics are a driving force for generating value from data, usually combining several visualization methods in order to maximize efficiency.

Web technologies are the most widespread approach for performing visual analytics in the context of smart cities and IoT networks (Mikusz et al. 2015). Stakeholders involved in smart cities, namely professionals and the public, are familiar with web-based applications, since they constitute an approach with which they come to contact on an everyday basis. A typical example of a web-based solution is SensMap (Simek et al. 2013), a framework focusing on the visualization of sensory infrastructure in indoor and outdoor spaces. Web analytics, including charts, graphs, diagrams, etc., are integrated into visualization systems, either as standalone systems or in combination with other visualization techniques (Zhang et al. 2012).

Cartography constitutes a traditional, yet effective and intuitive technique that displays information in accordance to its spatial characteristics. Colpaert et al. (2016) use geospatial visualizations of location logs in order to identify travel patterns. Even though cartography can provide a quick area overview, the necessity often arises to display additional information. In the domain of IoT visualizations, Blackstock and Lea (2012) combine maps and web analytics in the form of dashboards, correlating the sensors’ values with their spatial distribution. Furthermore, Ioannides et al. (2013) reconstructed cultural heritage content in 3D spatial reconstruction plus the time.

Apart from traditional 2D interfaces, visualizations using 3D environments and virtual reality are being increasingly adopted by researchers in the context of urban planning (Billger et al. 2016). Visual analytics are proposed by Batty and Hudson-Smith (2014), combining geospatial data with visual analytics. Apart from chart representations and cartography views, 3D environments visualize urban areas in the context of aiding the process of urban design (De Amicis et al. 2009). Another interesting approach is TrajGraph (Huang et al. 2016), which combines graph visualizations and GIS data, generated by drivers in order to create a visual analytics system for studying and planning urban networks.

Virtual reality (VR) is described as a 3-dimensional, computer-generated environment, which can be explored and interacted with by a person (Burdea et al. 2012). The related technology has been around since the 90s, but has not yet acquired widespread popularity. Since 2010, with the vast technology outburst of the 21th century, large tech corporations such as Google, Facebook, Sony, Samsung, etc. started to invest in this technology and making it publicly available. Despite this, published implemented applications of Big Data visualizations in VR are not common in academic literature. Head-mounted devices, such as Oculus Rift (Desai et al. 2014), are widely employed nowadays for virtual reality applications. Several approaches in literature employ virtual reality head mounted displays for specific purposes, e.g. discrete event simulation (Hutabarat et al. 2016). Another approach, which does not require head-mounted devices, is rooms equipped with motion sensors, such as the CAVE system (Torres and Palafox 2015). This approach, however, requires large space, costly installations and is difficult to be employed on a mass scale.

Olshannikova et al. (2015) present state-of-the-art issues and challenges in terms of Big Data visualizations with augmented and virtual reality. Helbig et al. (2014) use a virtual reality environment to visualize massive weather forecasting data. Another interesting approach is the immersive visualization of a landscape in Mars (Donalek et al. 2014), augmented with data describing the surface characteristics. Another domain in which virtual reality is employed is visualizations for Geographic Information Systems (GIS), which can distribute large data volumes in accordance to their spatial characteristics (Wang et al. 2015). Li et al. (2016) combine web based and VR visualizations in order to perform traffic and transportation analysis. Finally, three dimensional environments are combined with gestural interactions in the domain of data center infrastructure management (Drossis et al. 2016), providing a solution to navigation and item selection in 3D visualizations.

Interaction with virtual reality environments is primarily performed via touchless interaction. The aim of human-computer interaction is to eventually reach a point, where interaction is performed in a completely natural manner. The appearance of the users’ hands within the virtual environment enhances the feeling of presence in this environment and control over the application (Tecchia et al. 2014). Hand tracking is mainly implemented using sensors, such as Kinect and Leap Motion (Marin et al. 2014). Hand gestures can be employed for actions like pan-and-zoom (Nancel et al. 2011), device control (Mauser and Burgert 2014), writing in the air (Zhang et al. 2013) and spatial navigation in 3D environments (Valkov et al. 2010). Finally, multimodal interaction (e.g. speech-gestures combination) (Piumsomboon et al. 2014), offers a mechanism, which combines different communication channels, in an effort to increase expressiveness and imitate human habits.

1.3 Smart City Visualization Application Domains

Enhanced visualization techniques for smart cities can assist municipality personnel that handle typical daily tasks, such as the monitoring and control of city infrastructures. At the same time, at a higher level, these can also assist decision makers to improve operational efficiency, reduce cost, and minimise risks in managing city’s resources and infrastructure. The following domains constitute indicative areas, where visualization can assist the decision-making process:

Security/Policing: visualizing predictive analytics has been used in several cities across the world to help predict where crimes are likely to take place through historical and geographical data (Leitner 2013). Such approaches have been proven successful in many cities in the US (Levine 2006). By employing the power of data, a city can better manage law enforcement by focusing on areas where police intervention is more likely to be required. This can result into safer cities, with more reliable and trustworthy law enforcement policies.

City Planning: data can facilitate efficient city planning by maximizing accessibility of certain areas or services, whilst minimizing the risk of overloading important elements of the city’s infrastructure. At the same time, we often see buildings being built in areas that seem suitable but can have a considerable effect on other areas, without this being taken into consideration during the planning process. By employing data and modelling, it is possible to map infrastructure and building to city loads with a higher degree of accuracy (Sunesson et al. 2008).

Transport: public transport organizations can utilise data and analytics to manage and verify public transport route planning. Such systems can allow public transportation to input data from events, so as to predict passenger flows and thus optimise their effectiveness (Pereira et al. 2015). By employing data throughout a transport network, a city can provide effective and flexible public transport, decreasing delays and increasing efficiency. In addition to the prediction of peak times for upcoming events, data analytics can be used for traffic monitoring, improving reliability and decreasing the probability of accidents.

Future proofing: often, when new areas are created or become popular, the infrastructure in place is not sufficient to sustain continued growth, a fact that can hinder further improvements in the area. Even basic services, like water and electricity, can be affected by a sudden influx of businesses or residents. Through the use of modelling and predictive analytics, it becomes possible for city planners to see where these areas of growth are likely to be located; and how large this increase will be (Lede et al. 2016). Services can then be upgraded to accommodate growing needs, facilitating thus smooth growth.

Sustainability: one of the keys to sustainability is monitoring and acting in order to keep a city’s resources at certain levels. Data is the most decisive factor here, as it allows for cities to monitor how their decisions are having a positive or negative result on the city as a whole. Being able to check and control the levels of pollutants can help with zoning, placing pollutants in areas of the city where they can do the least harm or helping them to reduce their harmful effects (Siregar et al. 2016). Monitoring also creates the opportunity to investigate which technologies work best in reducing pollution; and what new innovations could be used in particular areas in order to prevent further environmental damage.

Photogrammetry and 3D city modeling: 3D city modeling and urban visualization, using the technology of photogrammetry, is one of the most growing research topics in digital architecture. There are many different methods of 3D city modeling, employed for various applications of 3D city models. The 3D city models are created not only for scientific visualizations, but also for architectural design evaluations. Another example of 3D modeling using photogrammetry tools is Esri’s CityEngine software.Footnote 1 Esri CityEngine is a three-dimensional (3D) modeling software application, developed by Esri R&D Center Zurich (formerly Procedural Inc.); and is specialized in the generation of detailed large-scale 3D city models in order to improve urban planning, architecture, and design.

1.4 System Architecture

The proposed approach aims at creating an adaptive, multi-purpose system, which is able to allow city stakeholders (e.g. administrative personnel, technical authorities, decision makers) to manage city data effectively and efficiently. Taking into consideration the potential benefits of VR towards smart city visualization, this paper proposes a VR-based approach, targeted to bringing immersive graphics and natural interaction experience to users. To this end, the system employs a three-dimensional virtual environment to visualize the city’s geographical context. The proposed approach renders the virtual city’s area, preserving the actual arrangement of buildings/infrastructure and visualizing their current state (along with other environmental data, e.g. indoor/outdoor temperature, noise level, air quality indicators); while it notifies users about potential anomalies. The visualization includes several condition indicators, updated in real time, as well as a colour-coding scheme for the current infrastructure condition, referring to a scale from normal to critical. Furthermore, the system supports on demand exploration of individual city’s data feeds (e.g., arrays of sensors data), providing detailed information about its condition for a specific time span, and combining historical analysis of previous values and the prediction of potential future state.

The visualization technique proposed in this paper can help stakeholders browsing for unknown relationships and question answering. In a planning environment (Suneson 2014) for example, the nature of two separate data sets might initially be fully understood, but not their interrelationship. A spatial analysis operation, such as visual overlay in a 3D environment, can be used to determine possible spatial data relationships. Questions such as “is there a spatial correlation?”, “what is the best site?”, or “what is the shortest route?” are typical of analytical uses of visualization tools and methods. Added value can be provided by performing post-processing on data flows to produce new knowledge regarding the visualized ecosystem. The system architecture of such a visualization framework is presented in Fig. 1.1.

Fig. 1.1
An illustration has three components a, b, and c under system architecture.

System architecture

Full size image

The main components of the framework are:

Devices: Sometimes referred to as “Things” or “end points”, devices can involve software or hardware components that generate data or measurements from the object they are monitoring and/or controlling. Some examples include temperature, voltage, humidity and machine rotation sensors. Sensors can be co-located as in the case of a wind turbine or a car, or can be part of another device (like the Smart Watch with biometric data), while some have control features like industrial valves.

Edge/Gateways: gateways are usually deployed closer to the edge (e.g., sensor node, some might actually be part of a sensor collection and are deployed with the sensor) and allow for the collection and selective transfer of data to the Hub, while providing some command-and-control functionality back to an automated device. This area is further expanding into what is called edge computing (Shi et al. 2016).

IoT data collector: an information collection daemon that can collect metrics from a wide array of inputs (ΙoΤ nodes) and write them into a wide array of outputs. It can be deployed in a distributed manner to collect, normalize, correlate, and aggregate metrics and events from sensors and other important data sources.

Time series database: A data storage service that supports high write loads and large dataset storage as well as preserves space through downsampling, automatically expiring and deleting unwanted data, as well as backup and restore. The developed component uses InfluxDB (Ahmad and Ansari 2017), allowing data analysis through an easy-to-use SQL-like query language.

Alerting and prediction system: This component can process both streaming data and batch data from the data store. The specific subsystem supports the plug in of custom logic and user-defined functions (e.g. custom rules, state alerting). Furthermore, machine learning libraries can be used to process data feeds and create alerts. In the context of alerting, it can support dynamic thresholds, perform pattern matching, compute statistical anomalies, etc., that can then trigger user-defined functions to form the basic IoT control plane. The component is designed to process streaming data in real-time and can be deployed across the infrastructure, both as a pre-processor downsampling and for performing advanced analytics before shipping the data to InfluxDB time series database.

Visualization system: in order to create the Virtual City, the visualization system communicates with the time series database and the IoT database to retrieve all the required data, analyse it and create the necessary data structures in memory for the scene data elements information representation.

1.4.1 Interaction

The interaction techniques for VR environments presented in this paper are based on the Leap Motion sensor mounted in front of an Oculus Rift, which displays a virtual world to the users. This setup allows free user movement in space, enabling them to turn their head towards any direction.

Gesture recognition is accomplished with the camera placed in front of the user’s head and therefore the user’s hands are never occluded by the user’s torso, which is a shortcoming for different setups where the depth sensor is placed in a static position. The user’s hands are rendered in the virtual world in a one to one mapping to the physical world, creating the feeling of a mixed reality environment, as the users perceive the hands that appear in their VR view as their own, and thus are confident that they have full control of the system. Several space navigation alternatives were examined. Based on previous work (Drossis et al. 2017), a closed fist gesture was selected for supporting navigation. Users can therefore move their closed fist towards the preferred direction and the camera will move accordingly. Even though the feedback of movement in the virtual space might be sufficient in the case of travelling in environments with nearby points of reference, such as the ground, walls or trees, when travelling at a distance from displayed elements, e.g. flying over a world, in space or in underwater environments, the movement speed and direction may be unclear. Therefore, an arrow indicating the user’s movement speed and direction is visualized over the closed fist that carries out travelling.

As far as rotating the viewport in the virtual three dimensional space is concerned, users are able to rotate their head and freely look in any direction in the virtual city. In terms of item selection, directing the pointer finger at an item is used for aiming at an element. The pointing direction is lighted and a circular cursor is placed on the interactive element, if any, to designate the ability to select it. Selection is accomplished through pinching, following the metaphor of clicking with a mouse.

1.4.2 Data Exchange

The initial population of the data structures that are required to visually represent the aforementioned information in the 3D scene is accomplished using standard HTTPS Calls in order to fetch initialization parameters. Communication messages use JSON syntax. JavaScript Object Notation (JSON) is a text-based data interchange format derived from the JavaScript scripting language that is very small and light weight. It is formatted as key-value pairs and is often lauded as being a lower-overhead to XML because it focuses more on content and less on formatting. This works to our advantage when we want to keep data interchange packets as compact as we viably can.

On the server side, parsing of JSON streams is faster due to small syntax, leading to faster responses. JSON server-side parsing is the strong point that indicates us to use the JSON on the server side.

An example response for two example buildings is presented below:

Code block 1 Initialization data exchange protocol

Each building or asset (JSON defined in POI/node/type) is described with its name (POIFootnote 2/name), GPS location (POI/longitude, nodes/POI/latitude). All the data feeds URL’s are defined in POI/node/data feeds and include the type of data and the HTTPS link to the resource. Data feeds also return JSON time series data. The next code block displays the data format from a sample temperature sensor (2 samples example):

Code block 2 Sample data from temperature sensor

1.5 The Virtual City Environment

The presented application aims at helping city’s stakeholders to get an intuitive overview of their city regarding its current/historical conditions of distributed IoT sensors. A hybrid approach, which combines a three dimensional environment and a visual analytics platform, adapted to a virtual reality environment is proposed. Therefore, users can act immersed in a 3D environment, interact with dynamic representations and still benefit from the statistical representation that visual analytics offer. Examples of statistical representations are presented in Figs. 1.2 and 1.3.

Fig. 1.2
A screenshot has a map with some pointed locations with real-time sensor measurements on Heraklion smart city web page.

POIs with real time sensor measurements

Full size image
Fig. 1.3
A screenshot has a graphical representation of an environmental data explorer on Heraklion smart city web page.

The environmental data explorer

Full size image

The VR application facilitates the inspection of the buildings and city’s infrastructure by the users and warns them, in an intuitive manner, about infrastructure state that needs further investigation, such as an anomaly regarding particular machinery (ex. pump machines) that may bring to surface malfunctions or degraded operation in the city. In addition, the users can inspect city’s assets, retrieving supplementary information and employing visual analytics on demand.

Our approach employs the map metaphor, as it is based on the concept of having a virtual geospatial visualization, which constitutes the foundation of user collaboration. Thus, the city’s virtual environment extends along a plane, which is placed on a virtual table, constituting a common ground on which users can make observations, reason on data, gain insights and ultimately draw conclusions that can assist decision-making.

The application currently supports up to two users, who view the virtual city from their own perspective, in accordance to their physical location and viewport. Each user can employ hand gestures in order to modify the virtual map representation, allowing common actions, i.e. panning and zooming. By performing a fist gesture on the virtual table’s perpendicular axis, the users can zoom in or out the presented map. In addition, the fist gesture offset along the table’s plane is applied for panning on the map and allowing the flow of city’s areas. Furthermore, the selection of any city asset is visible to both users, allowing each user to share his/her findings and collaborate with others.

The main screen of the Smart City 3D Visualization application, which comprises a virtual representation of the city and the basic interactive UI components, is depicted in Fig. 1.4. All buildings are grouped and displayed as 3D objects according to their physical location in the city’s space. The virtual city is constructed by using digital models of urban areas that represent terrain surfaces, sites, buildings, vegetation, infrastructure and landscape elements as well as related objects (e.g., city furniture) belonging to urban areas. The city’s sensory infrastructure is visualized as a network of interconnected points of interest (POIs). Each point of interest holds its own minimal representation (Fig. 1.4), which can be extended to host a short description of its current status.

Fig. 1.4
A map of I o T live data exploration has some locations highlighted on it.

IoT live data exploration

Full size image

Users can select the various points of interest by either pressing them with their virtual hand representation or by employing point-and-click. Upon the selection of a POI (e.g. a city asset or a specific building), the visualization changes and extended information per asset is displayed. The close up view (Fig. 1.5) contains historical data information, which is automatically updated when the IoT infrastructure provides new data. Historical data is presented using line charts in a spherical view, resulting in the enclosure of the user in a spherical display of information. The presented chart representations can be altered by hand gestures, as users are able to pinch and drag diagram axes in order to extend the viewport of the provided data.

Fig. 1.5
An illustration has a 3 D generated hand model and some graphical representation with temperature values of a P O I.

Interaction with temperature values of a POI in close-up view

Full size image

Apart from representing distinct values about city assets, the presented application facilitates the visualization of trends in data. The 3D rendering process can be altered to apply advanced rendering techniques (e.g., global illumination and shadow calculation, illustrative rendering, heat mapping) in order to visualize geospatial distributions of data. Such representations can be useful for providing macroscopic inspection of metrics, which share similar values in accordance to their distribution in space, such as air quality, pollution indicators and temperature (Fig. 1.6). These visualizations can facilitate pattern recognition and correlation of metrics, e.g. wind direction with air pollution, unique micro-climates within the city, etc.

Fig. 1.6
An illustration has a 3 D generated hand model pointed toward a 3 D generated heatmap with some areas highlighted.

Sample data from temperature sensors

Full size image

1.6 Evaluation—Validation

The visualisation infrastructure presented in this research paper was technically validated at the premises of the FORTH-ICS AmI research facility. The validation was carried out in an iterative process. Initially the visualisation framework was instantiated on top of a city model, generated through Google Maps and with offline dummy datasets. This step was required so as to test the capability of the framework to visualise information on top of city models; and at the same time validate the visualization structures and algorithms employed.

The second step involved the validation of interaction. For the purposes of this experiment, a setup was made involving one high end desktop PC (Intel i7 processor @ 4.6 GHz, 8 GB Ram, NVidia GTX 680), an Oculus Rift device, a Leap Motion sensor mounted on the HMD and a leap controller. During validation with a limited number of expert users (members of the Human Computer Interaction Laboratory of the Institute of Computer Science of FORTH with expertise on VR), interaction was fine tuned for both free hand interaction and controller based interaction in the virtual world. The experiment involved both visualisation tasks and navigation on the city model for real time data exploration.

The final validation step was conducted by integrating data from the Heraklion IoT and Open Data ecosystem, currently developed under a Programmatic Agreement between the Municipality of Heraklion and FORTH-ICS. This final step was essential in order to discover the usability of the framework in an actual smart city platform. The third step was conducted with the same user set of the second experiment. The goal was to cross-correlate the experience on dummy data sets with the one gained when exploring an actual smart city infrastructure.

1.7 Conclusion and Future Work

Blending VR with data visualization brings out new ways of creating added-value services across a wide range of sectors, from industry to services and utilities. Effective data visualization remains one of the key challenges in the era of Big Data—a cognitive bottleneck on the path between data and discovery. Our goal is to maximize the intrinsic human pattern recognition (or visual discovery) skills through the use of emerging technologies, associated with immersive VR.

The biggest difference between virtual reality and flat, map-based data visualizations is that you can be placed “inside” the data, rather than above it. We’ve experienced an unexpected intimacy from these visualizations, along with a sense of scale… The subjective point of view is also more cinematic than a map, making it potentially more powerful for storytelling.” Brian Chirls, Creative Developer.Footnote 3

Regarding future exploitation, the proposed framework will be further extended to support the full set of data generated by the Heraklion IoT and Open Data ecosystem, together with data stemming from other municipal services. At the same time, the final prototype will be evaluated by actual users, stakeholders of the municipality of Heraklion (administrative personnel and decision makers), both in terms of visualization and interaction.