Keywords

10.1 Introduction

Natural and man-made disasters are unpredictable and difficult to manage and cause considerable devastations to city infrastructures and loss of human lives. According to CNN Money [22], natural disasters caused a global cost of 175 billion dollars in 2016, for which only 30% (about 50 billion) were insured. This loss was recorded as the highest in the past 4 years. Furthermore, it is also important to note that in 2016, North America suffered a total of 160 disaster events, which is also reported as the highest number ever since 1980 [22].

Disaster management is often divided into four well-known phases: prevention/mitigation, preparedness, response, and recovery. Prevention or mitigation methods are developed through the different types of approaches, such as disaster risk analysis, vulnerability analysis, resource management, and planning. Most disasters may be impossible to prevent but can be effectively mitigated by analyzing the situation and making the right decisions. Disaster data modeling is helpful to understand the risks and to create an estimate for the losses given a certain scenario. Modeling such a complex system requires the collaboration of professionals in various fields [27]. By analyzing past events, data can be simulated, and possible future scenarios can be assessed.

Multimedia data includes text, image, audio, video, etc. In disaster management, various types of data might come from different sources, and thus effective data fusion and multimodal data analysis approaches are important and challenging to extract useful knowledge for a better understanding of a disaster. Furthermore, visualization is helpful to represent the combination of several multimedia data types in a way that is easier to understand and interpret. Figure 10.1 shows a high-level framework of multimedia data management for disaster situations.

Fig. 10.1
figure 1

Multimedia data management framework for disaster situations

Full size image

This paper is organized as follows. Section 10.2 discusses various multimedia data types in disaster management. Several well-known data analysis techniques are introduced in Sect. 10.3. A case study introducing the rendering of a storm surge animation in a 3D environment is given in Sect. 10.4. Finally, Sect. 10.5 presents the outlook and future direction.

10.2 Multimedia Data in Disaster Management

The process of data collection and management to provide the aid, relief, and response in disaster situations has always posed a challenge to public emergency services and the disaster management community, due to (1) time constraints; (2) the scattered nature of data sources, owners, and types; and (3) speed of access to data, as pointed by Meissner et al. [15].

Different multimedia data types and sources related to disaster management are listed below.

  • Visual Data With the advent of low-cost and high-quality imaging devices, at both the consumer and professional levels, the speed and rates of data generation have spiked. The analysis of visual data involves multiple research disciplines, ranging from geospatial analytics to data management and knowledge representation [13]. Recent efforts in enhancing the capture of both image and video, during any phase of the disaster management cycle, have seen a great benefit from the employment of unmanned aerial vehicles (UAVs), specifically from (a) the lower costs of UAVs, (b) its increasing capabilities to capture high-resolution images, and (c) the large amounts of data and metadata generated. In [19], an image dataset is built from the key-frame samples of disaster-related videos on YouTube, which was used as the input of the training and testing of their proposed weighted discretization analysis framework. A very important characteristic of the datasets built from visual data is mentioned in that paper, namely, their imbalanced nature. In [29], Yang et al. take a step further and classify images along with the textual information. Additionally, data from situation reports and user feedback are also taken into account.

  • Sensors Data The rapid and explosive growth of mobile phones, wearable technology, and Internet of things (IoT) devices has established the collection and processing of data from sensors as an ongoing discipline and research trend. Sheng et al. [23] present one of the first general approaches at leveraging the sensors’ capabilities in mobile phones. An important topic in this research is the management of energy efficiency through collaborative sensing as a means to reduce redundancy, thus reducing the energy consumption. A more recent and disaster-management oriented approach is proposed in [5]. By combining the sensing capabilities of Wireless Sensor Networks (WSNs) and the visual capturing capabilities of multi-UAVs systems, applications in disaster management can range from early detection systems to damage assessment and rescue missions, to name the most important ones. In their approach, WSNs act as the detection systems that trigger the launch of diverse types of UAVs in the wake of a disastrous event.

  • Geographic Data Geographic Information Systems (GIS), like all technology-supported systems, have come a long way from their inception and underutilization days due to technology limitations [30] to today’s pervasiveness. The combination of low cost, mobility, and portability of new devices with geographic capabilities, as well as user-generated content, willingly or unwillingly, has built enormous databases for different purposes. In 2007, Goodchild [7] coined the term volunteered geographic information (VGI) to reference all the voluntarily user-generated geographic and geospatial content. Since then, almost all related literature and research in the area have used this term. A broad survey by Granell & Ostermann [8] about the literature and research where VGI is paramount in disaster management found that (a) most of the studies still focus on data and how it should be collected and processed, (b) Twitter is used as a major data source in data-centric research, (c) reliance on Twitter as a unique data source might hinder the validity of the results, and (d) few studies combine VGI with data from official sources.

  • Social Media Data As for social media data, thanks to the pervasiveness of social media networks, blogs, content generation tools, and photo and video sharing applications, users have become active entities rather than passive participants [6]. Research focus related to disaster-related data originating from social media streams is heavily inclined toward two main trends: (1) the extraction of useful pieces of information and data from the diverse social media networks and (2) the crowdsourcing of knowledge during disaster events. Imran et al. have done extensive research about extraction in [11]. However, as data from social media can sometimes be inaccurate, extemporaneous, malicious, and/or noisy, Planella Conrado et al. [3] have proposed a framework to ensure the quality of data and information at the user and emergency field-responder levels, with the participation of the informed actors and based on near real-time verified data to support the decision-making processes of the non-informed participants.

  • Storing and Sharing Data There are several advantages and opportunities that the current state of technology can provide: (1) large amounts of data can be generated by the ordinary users, and (2) current technology provides a means to store and process all of these data. Grolinger et al. [9] have proposed a Knowledge as a Service (KaaS) framework as a means to handle a near complete integration of different data sources. By leveraging the advantages of the relational and NoSQL databases, along with diverse cloud technologies, the proposed framework can be capable of (a) satisfying some of the requirements of data for disaster management systems outlined by Meissner et al. [15] and (b) dealing with some of the characteristics noted by Hristidis et al. [10].

10.3 Multimedia Data Analysis in Disaster Management

A key aspect of disaster management systems is how to analyze the data generated in disaster-based situations in an efficient and effective manner. Data mining and machine learning techniques can be utilized to address the challenges in disaster data analysis. These techniques can be divided into several main steps, such as preprocessing, feature or attribute analysis, learning, and post-processing.

  • Preprocessing Similar to general multimedia data, real-world disaster data can be extracted from multiple heterogeneous sources. Therefore, it is noisy, incomplete, skewed, and inconsistent in nature. In reality, it is impractical to use such complex raw data as the input of the analysis algorithms, and thus careful preprocessing techniques are needed. In general, preprocessing includes, but not limited to, data cleaning, normalization, formatting, reduction, transformation, and missing value interpolation [10]. Jain et al. [12] present a real-time disaster data mining framework based on social networks. Since the extracted tweets include many irrelevant information, data preprocessing is essential to save computational power and to generate more useful data. In particular, language and geographic information are utilized to filter and clean this large and noisy data. Preprocessing is also an important component of the Florida Hurricane Loss Model (FPHLM) [27], a public hurricane model estimating loss costs for personal and commercial residential properties, because this model receives different data formats, often containing noisy and missing values, from the insurance companies. Although preprocessing is a key part of every disaster management system, it was barely discussed in the literature. To the best of our knowledge, there is no general tool or application to automatically clean and format the disaster multimedia data. This can be due to the heavy domain knowledge needed for each disaster-based system which cannot be generalized for other situations.

  • Feature Analysis Features or attributes are used to discover the knowledge in a dataset and characterize the instances in it. Researchers extract and select useful low-level and mid-level features from the data to find its high-level contents systematically. As disaster data may include different data types, including visual, aural, textual, GIS, etc., it is challenging to extract multimodal discriminative features from the data instances. Low-level visual features such as color, shape, texture, and wavelet, as well as textual features, are extracted and combined for situation report enhancement in [28]. Geospatial information is another important feature widely used in current disaster management systems. A real-time crisis mapping framework is developed to geoparse tweet contents [16]. This framework uses the existing geospatial tools (e.g., OpenStreetMap) to extract street-level, buildings, regions, and local features. Deep learning is a new but powerful method which can be used for feature extraction from raw data. To automatically extract rich features, transfer learning which employs the existing deep learning models (pre-trained on very large datasets) on new data can be leverage. This helps the researchers discover the high-level abstractions (meaning) even with the limited data. Deep neural networks are also used for social media analysis for crisis response and disaster management [18].

  • Machine Learning Machine learning is the process of discovering, predicting, and learning from data or through experiences. In disaster and emergency management, data comes from a variety of sources and different kinds of knowledge may be needed by different users. Thus, machine learning and data mining algorithms may involve different tasks including clustering, association rule mining, trend analysis, and classification, to name a few. The major challenges in this phase include the following [10]: (1) disaster data is big and heterogeneous in nature, (2) it includes noisy and uncertain information, (3) its distribution is skewed and imbalanced, and (4) the disaster applications are domain specific. Therefore, conventional machine learning techniques cannot easily handle such complex, multimodal data in an efficient manner.

    It has been shown that multimedia data mining can handle the challenges in disaster management applications. In [19], a disaster-based video concept detection approach is proposed using weighted discretization multiple correspondence analysis (MCA) which enhances the correlation between the targets and the feature values. MCA is also utilized in another study [28] where a hierarchical image classification algorithm is proposed for disaster response situations.

    Nowadays, deep learning has shown its great potential in different applications such as computer vision, natural language processing, and speech processing. In recent years, the advantages of utilizing neural networks [25] and deep learning [20, 24] in disaster management systems are discussed. For instance, Song et al. [24] present a system which detects the human behavior and mobility in emergency situations using deep learning. Machine learning and deep learning also play important roles in robotics [14]. In these days, trained robots can save lives and prevent disasters, but the main challenge remains in analyzing the data in real-time for disaster recovery situations [2].

  • Post-processing The results generated by knowledge acquisition algorithms (e.g., neural network, decision tree, etc.) need to be post-processed to be useful and appropriate for the user and customer views. Post-processing, an important component of data mining, includes knowledge filtering and pruning, interpretation, evaluation, and integration. As a disaster happens, a huge amount of data (e.g., social media, videos, mobile data, reports, documents, etc.) is generated and then analyzed by the data mining techniques. Thereafter, efficient techniques are required to filter out and summarize the results. In addition, the results from multimodal sources need to be integrated and fused [28]. Similar to preprocessing, post-processing plays a critical role in FPHLM [27].

  • Tools and Applications The uses of tools and applications in disaster management have been evolving from time to time. This can result from technological advancements and lifestyle changes, but the objectives remain the same. The goal is to help essential personnel to be quickly aware of the current disaster situation, efficiently respond to different requirements, and make the optimal decision in the shortest time. In Table 10.1, several tools and applications are presented and categorized by different characteristics.

    Table 10.1 Tools and applications in disaster management system
    Full size table

10.4 A Case Study: 3D Storm Surge Impact Animation

Visualization methods serve to assess complex disaster events in an interactive manner. The 3D Storm Surge Impact Animation [21] is a visualization environment which uses GIS data to simulate the impacts of a storm surge near real-world coastal areas. More specifically, the 3D model is a visual representation of South Miami Beach. The animation is built using a 3D game engine, Unity,Footnote 1 which opens many possibilities for its deployment, namely, its ability for cross-platform support for the popular operating systems (e.g., Windows, Mac, and Linux). The use of the GIS data makes it possible for the model of the city to accurately match a real-life environment. To produce the terrain that depicts an accurate visualization of South Miami Beach, the Light Detection and Ranging (LiDAR) downloaded from The National Oceanic and Atmospheric Administration (NOAAFootnote 2) website is used. LiDAR is a remote sensing technology which produces point clouds, and each point represents the elevation at a specific location. From the LiDAR point cloud data, the bare-earth points can be extracted and a digital elevation model (DEM) can be created, which is a grayscale raster where each pixel contains the height information for each location. Unity makes it easy to connect the animation with the Integrated Computer Augmented Virtual Environment (I-CAVEFootnote 3), a visualization and research facility, ideal for presenting 3D virtual environments. I-CAVE gives users the capability of navigating through the terrain as an immersive experience. The resulting visualization, as shown in Fig. 10.2, can be used in studying the effects of storm surge in a real-world environment.

Fig. 10.2
figure 2

Debris are produced by broken buildings and trees. The model includes different types of debris: pieces of broken wood, building rubble, and torn tree branches, which are affected by the wind and the surge of the waves

Full size image

The following is a list of different parameters that the user can change to create different types of storm surge scenarios.

  • Wind Scale: When the user selects the category of the hurricane, the intensity of the wind is set according to the Saffir-Simpson hurricane wind scale.Footnote 4 Unity’s built-in wind zone component makes it easy to set the parameters for the changes in the wind.

  • Rain Intensity: The rain animation was created using the Unity Particle System. A user is able to change the parameters that affect the visual of the rain. Such parameters include the intensity of the rain and the force factor of the wind on the rain.

  • Wave Intensity: Waves are able to flood the land, smash the trees and infrastructures, and carry scattered debris to multiple directions.

  • Debris Properties: A parameter that a user can set for debris is the average weight which determines how much the scattered pieces can move according to the force of the wind and how much damage it can produce when it hits a building.

  • Tree Bend & Break Factor: The tree bends according to the effects of the wind. Under certain conditions, the wind can be strong enough to break a branch from a tree.

10.5 Conclusion and Future Directions

With the advancement in multimedia and data mining techniques, as well as the proliferation of smart technologies including mobile, wearable devices, and big data, disaster management systems have become more intelligent and efficient. This paper summarizes the state-of-the-art techniques in multimedia data management for disaster situation awareness. Specifically, it discusses how disaster data are obtained from various sources such as social media, sensors, and videos. Multimedia data analysis for disaster management, including new techniques of preprocessing, feature analysis, machine learning, and post-processing, is presented. In addition, several tools and applications in this area are introduced. Finally, a case study is presented to introduce a 3D storm surge impact animation. Despite the great potential of multimedia data management, there are very few approaches leveraging it in disaster recovery systems.

It is foreseeable that multiple disciplines will intensively work together and be more aggressively engaged in facilitating management in all disaster phases. Tremendous progresses have been made in artificial intelligence (AI), which make robots to be extremely helpful in tasks like searching, rescuing, and inspecting the disaster site without additional casualties. Multi-agent system has also shown promising performance in emergency management tasks. Each agent is independent and versatile and is able to allocate the resource, identify the optimal route, and respond to potential obstacles. All the mentioned characteristics can perfectly fit into any disaster scenario.