4.1 Introduction

With the rapid development of computers, information, and communication technologies, and the increasing availability of mobile phones and social media sources, our living space has been transformed from physical space into a form shared by both physical and virtual spaces (Yu and Shaw 2008; Shaw and Yu 2009). The convergence of geographic information systems (GIS) and social media facilitates the understanding of spatial dynamics of human behavior and societal transformation (Sui and Goodchild 2011). There have been extensive studies on how information (e.g., media contents) diffuses and spread in a social communication network (e.g., Twitter and Facebook). Researchers have developed new theories, methods, and tools for exploring patterns and knowledge hidden in complex human dynamics and social interactions within the context of today’s mobile and big data era (Gao et al. 2013a; Liu et al. 2015; Hu et al. 2015; Shaw et al. 2016, Ye et al. 2016a). Such spatial, temporal and social dynamics can also help discover and understand the rise and fall of emerging topics and events of the society (Shahaf et al. 2012; Peuquet et al. 2015), as well as reveal the digital divide, the physical divide, and the social segregation in developing countries (Amini et al. 2014; Gao et al. 2017). Spatiotemporal visualization techniques are effective in detecting human activity patterns over space and time, while social network analysis is good for exploring graph structures and interactions among individual nodes and groups. Integrating these two types of methods can facilitate the exploration of complex social networks in space and time, and can help discover hidden spatiotemporal and social connections. Luo and MacEachren (2014) proposed a theoretical framework to integrate geographical context, network context, and societal context to understand the geo-social interactions in both physical and virtual spaces. However, few studies so far have made efforts on developing integrated visualization approach and quantitative metrics to measure such interactions in a spatio-temporal-network synthesis framework.

Geo-social network visualization can be classified into two major groups: the first group focuses on spatializing network representation; the second group focuses on exploring geo-social relationships and interactions. The first group tends to integrate network representation into geographical space that has been widely applied to research related to spatial trajectory and interactions, including migration flows (Tobler 1970; Guo 2009), human travel behaviors (Kwan 2004; Gao 2015), transportation flows (Huang et al. 2012), and so on. These studies consider spatial trajectory and interaction from an exclusively spatial perspective without taking their social relationships into account (Shi et al. 2015). Increasingly interconnected societies through technical-social systems (e.g., social media) have called on the necessity of combining social network analysis with spatial analysis (Luo et al. 2011; Luo and MacEachren 2014; Andris 2016; Steiger et al. 2016). Lee and Kwan (2011) integrated 3D space-time paths and ring-based visualization of social networks for analyzing socio-spatial isolation information. Luo et al. (2014) developed a visual analytics tool, the GeoSocialApp which supports the exploration of geographical, social network, and multivariable attribute space. Yin and Shaw (2015) proposed a spatio-temporal analysis framework for exploring physical separation and social interactions at the individual level. Luo (2016) developed a visual analytics tool, the GS-EpiViz, which allows the exploration of human geo-social interaction patterns to design effective disease control scenarios. These studies proposed several useful visualization framework or analytical approaches for simultaneous discovery of geo-social patterns in physical space or in both physical and virtual spaces. Challenges still remain, however, in quantitatively exploring and measuring relationships between physical movement and social closeness in virtual space.

Researchers have also been actively investigating the spatiotemporal effect in the information diffusion process across a wide range of disciplines (Tsou and Leitner 2013; Ye and Lee 2016). The processes of information diffusion over various network configurations can be analyzed following the approaches in the spatial diffusion studies initiated by the pioneering work of Hägerstrand (1967). Gregory and Urry (1985) argued that Hägerstrand’s models cannot deal with spatial diffusion via social networks as well as the associated conflicts and resistance. Morrill et al. (1988) subsequently categorized quantitative models of spatial diffusion processes into stochastic models and deterministic models. Cao et al. (2012) traced the pathways of retweets on a spatial hierarchical layout. Garcia and Wimpy (2016) explored the proliferation of communication technology on information diffusion and violence spread across neighboring states. Spatiotemporal visualization can be applied to intuitively reflect how people share ideas, disseminate information, and communicate with each other, a dynamic process whereby ideas spread through communication networks over time (Tsou et al. 2013). He and Chen (2016) investigated the spatiotemporal text data with visual filters. Novel visualization techniques facilitate the iterative and interactive exploration of the dataset in order to gain deeper understanding of the origination, propagation, and clustering of information (Ye et al. 2016b).

In this research, we propose a conceptual framework for spatiotemporal and social network visualization in a three-dimensional context and present several new quantitative metrics for measuring underlying dynamic interactions among entities. The proposed framework aims to help better understand spatiotemporal patterns of human dynamics and social interactions over both physical and virtual spaces simultaneously, as well as explore how emerging events trigger spatial-temporal-social interactions and information diffusion from a process perspective. As a proof of concept, we demonstrate the proposed framework with a collection of geotagged tweets in the ArcScene software. This research provides new insights on integrating multidisciplinary knowledge to explore human dynamics in a broader way.

4.2 An Integrated Spatio-temporal-Network Framework

Here, we introduce a new spatio-temporal-network conceptual framework (STN). As shown in Fig. 4.1, it is an integrated framework which consists of the spatial, temporal, and social context of a subject or an event (labeled as Nodes). The spatiotemporal coordinates locate where and when a subject or an event is, while the network linkages (labeled as Edges) help know with whom a subject is or where the geo-social impacts of an event are and how the information spread across different people and places via physical movements or the social network connections.

Fig. 4.1
figure 1

An integrated spatio-temporal-network conceptual framework

Full size image

There are three types of edges in the STN. First, the physical edges connect individuals’ movements in the physical space (i.e., the space-time path, STP). Second, the social edges represent the individuals’ social relationships (e.g., friends and colleagues) or social interactions (e.g., emails, phone calls, retweeting behaviors, and other social media communications) in the virtual space. The subtypes of social edges could vary in different contexts in which at least one or multiple subtypes of social edges could be added in a STN framework instance. Third, the physical-social edges connect the individuals’ physical locations to their social activities. The third type of edges may or may not be visible in a STN depending on whether the nodes belong to the same type. For example, if the nodes are all individual persons in a STN, we may hide the physical-social edge between an individual and his/her tweet message but only visualize how individuals get connected through retweeting behaviors in the virtual space.

The geographic trajectory nodes which follow a spatiotemporal sequence form the classic STP may form a STN, in which subjects in spatial trajectories are related because of social interactions for a given period triggered by a specific event. The topology or attributes of nodes in such spatial networks can change with time. Information can diffuse through connected nodes in the social network no matter whether nodes are adjacent or not in the physical space. However, the paths (sequential connections) among those spatiotemporal nodes need to obey the time order of the appearance of network edges, which share similar characteristics and inference issues of temporal graphs (Kempe et al. 2000; Holme and Saramäki 2012). The first-degree nodes which are directly connected to the target subject receive the first-hand information, while the second-degree nodes directly connect to and get information from the first-degree nodes, and so forth; the STN has the capability to visualize the physical space-time trajectories and their corresponding Nth-degree virtually connected nodes simultaneously. One potential challenge would be to deal with a large number of individuals and their activities in both physical and virtual spaces. Several visualization techniques can be considered for better visualizing large-scale edges and flows, such as the spatial generalization and aggregation method (Andrienko and Andrienko 2011), the geometry-based edge-clustering framework that can group edges into bundles to reduce the overall edge crossings (Cui et al. 2008), and the hierarchal clustering flow visualization approach (Zhu and Guo 2014).

Human behaviors and information spread by social media has greatly changed the traditional sense of decision-makings. Such a STN can help us visualize and analyze complex spatio-temporal-social interactions. In social networks, the persons who have active interactions in the virtual space may have also had certain connections in the physical space previously. However, how long will the previous physical connections continuously bring virtual interactions? This may depend on the semantics of a person’s activities in physical place. For example, attending a conference at a city may have different time decay effect compared with attending college at the same city. The STN may help us visualize and quantify such time decay effect. In addition, this framework can also help with the responses of disaster alerts and evacuation orders. For example, if a wildfire starts, the emergency response staff needs to take action in both physical space and virtual space to announce the evacuation order and the locations of new evacuation shelters, and to reach more people before any further disaster happens and grows.

More specifically, the proposed STN framework may help answer the research questions such as

  • How does information diffusion over physical space and virtual space behave differently in normal and crisis situations?

  • How do individuals’ movements and activities in physical space interact with their social ties in virtual space?

  • Which nodes are more influential on specific region and time period?

  • What strategies can organizations or governments use to leverage social media and real work channels for decision-makings?

  • Can this framework help social scientists trace, monitor, and analyze human dynamics in different domains?

  • Which context (spatial, temporal, or social context) is more important to influence information diffusion in different application scenarios?

  • How to predict the spread (speed, scale, and range) of social media messages in different spatial-temporal-social networks?

Here, we just list a few potential research questions which could benefit from the proposed STN conceptual framework. Our aim is to stimulate interdisciplinary thinking on research challenges and applications which involve the spatial, temporal, and social aspects.

4.3 Case Study

As a pilot study, we analyzed the recently finished Olympic Games in Rio 2016. This event had drawn attention from people all over the world and generated a large number of topics on social networks, such as Twitter, which can help study the interactions in both physical and virtual space. Often, athletes and reporters in Rio de Janeiro tweeted during the event and interacted with Twitter users from elsewhere in the world by retweeting, commenting, and liking their posts. In particular, we chose to study one of the most influential athletes during this event, Michael Phelps, because his tweets have created a network of connections with people from not only the U.S. but also other countries in the world.

According to Michael Phelps’s Twitter account, there were 12 tweets during the time he was in Rio between August 2nd and August 17th. In order to illustrate our STN framework for supporting physical-virtual interactions, we analyzed 3 of his posts: one posted on August 14th was after winning the gold medal of the 4 × 100 m medley relay, which had the most likes among those 12 tweets; another tweet on August 13th was after he was defeated by a young Singapore athlete Joseph Schooling; and one more post was after he returned back to the U.S. Moreover, we also took Schooling’s post on August 15th into account, which was a retweet from Phelps’s first tweet on August 13th. The detailed descriptions of these tweets can be found in Table 4.1.

Table 4.1 Detailed information of four analyzed tweets
Full size table

We are interested in analyzing whether there is any geographic discrepancy between the interactions in physical space and those in virtual space over a period of time. Therefore, as picking sample points, we randomly selected 1% of the total tweets. For example, for the tweets on August 14th, we chose 30 posts responding to Phelps’s tweets which contain geographic locations and timestamps. From about 2000 retweets of Phelps’s tweet on August 13th, we also randomly picked 20 posts related with defined geographic locations and timestamps as well. The collected data were imported into the ArcScene to generate 3D model for visualization.

As shown in Fig. 4.2, this 3D space-time-network model created by ArcScene uses x and y coordinates for representing the space, z values for representing time. Points (nodes) represent tweets and polylines for representing either physical movements or social network connections. We acquired x and y coordinates by geolocating tweets to the city-scale. To better illustrate the time in the model, we converted time into minutes. From the visuals, we can see that people do not only interact with people on social network nearby in the physical space, but also interact with those who are far away from them in the physical space. Therefore, when exploring those connections after August 14th, we find out that interactions mostly happened in the United States and few connections with people in Rio or other cities in the world. Moreover, another event relating to Joseph Schooling showed a similar pattern. However, since Schooling is from Singapore, after he interacted with Michael Phelps on Twitter, there were some additional connections to Singapore. Also, as a college student in the University of Texas at Austin, Schooling connected with users near that region as well. In other words, a person’s connection with people at a physical place in an earlier time period often brings interactions with the same group of people through virtual space in a later time period.

Fig. 4.2
figure 2

Spatio-temporal-network visualization of geo-social interactions on Twitter

Full size image

Moreover, Michael Phelps’s tweets have shown a movement trajectory through a scale of time. He moved from Rio back to the U.S., which is indicated by the white arrowed line on the map. As an influential user in virtual space based on the order of time, Phelps’s tweets showed a sequence of path with first degree connections that consequently lead to second degree nodes. On top of that, there were more connections generated from users outside of the U.S. during the Olympic, comparing with those after he went back to the U.S.

As for the temporal distribution, we can see that there were more immediate interactions with users living in the U.S. Those lines seemed to create a flatter surface. One possible reason was that there were more users in the U.S. who were more active and paid more attention to Phelps’s events. However, as for the connections between users in Asia with Schooling’s tweets, they went through a longer time to respond to his post because of time-zone differences, while links to the U.S. showed a similar pattern as that of Michael Phelps’s interactions. In short, this case study demonstrated that the STN framework could help understand how events trigger spatio-temporal-social interactions.

4.4 Vision for Quantitative Analytical Metrics

The value of this new conceptual STN framework is in not only considering the art of visualization for designing but also supporting potential development of quantitative analytical indicators for measuring complex interactions among subjects in physical and virtual spaces. Here, we present some of those possible metrics with the STN framework.

  • Spatio-Temporal-Network Impact Extent (STNIE)

In order to know the spatial impact extent {(Xmin, Ymin), (Xmax, Ymax)} of a message or an event within a given time period (Tmin, Tmax) in a social network, an indicator of the spatio-temporal-network impact extent (STNIE) is defined as:

$$STNIE = \{ ({\text{X}}_{\hbox{min} } ,\,{\text{Y}}_{\hbox{min} } )_{\text{ti}} ,\, ( {\text{X}}_{\hbox{max} } ,\,{\text{Y}}_{\hbox{max} } )_{\text{tj}} |{\text{T}}_{\hbox{min} } \le {\text{ti,}}\,{\text{tj}} \le {\text{T}}_{\hbox{max} } )\}$$
(4.1)

The STNIE will be a space-time box in the integrated STN visualization framework. It can help quantify and compare the social impact of an individual node in space and time.

  • Spatio-Temporal-Network Impact Center (STNIC)

In order to identify the central impact location at a given time period for a given message or an event in the social network, we present an indicator so-called spatio-temporal-network impact center (STNIC). It can have different implementations, such as the weighted mean or the weighted median. The coordinate for weighted mean STNIC is defined as:

$$\bar{X} = \frac{{\mathop \sum \nolimits_{i = 1} W_{i} * X_{i} }}{{\mathop \sum \nolimits_{i = 1} W_{i} }} ,\quad \bar{Y} = \frac{{ \mathop \sum \nolimits_{i = 1} W_{i} * Y_{i} }}{{\mathop \sum \nolimits_{i = 1} W_{i} }} ,\quad \bar{Z} = \frac{{\mathop \sum \nolimits_{i = 1} W_{i} * T_{i} }}{{\mathop \sum \nolimits_{i = 1} W_{i} }}$$
(4.2)

where W i is the weight for each impacted node (X i , Y i , T i ) in the space-time coordinate system. The value of W i could be the importance of a subject if a node represents a person or could be the number of impacted entities attached to a location if a node in the network represents a place. The weighted mean STNIC might be an arbitrary location which does not exist in the original nodes. Alternatively, by applying the weighted median, a node in the network will be selected as the STNIC. It can be further used for tracking dynamic changes in the spatial distribution of information spread in the virtual space by analogy to human migration center in the physical space (Plane and Rogerson 2015). In our case study, the STNIC for Phelps’s tweets locate in the U.S. while that for Schooling is in Asia.

  • Spatio-temporal-Network Distance (STND) and Efficiency (STNE)

In classic social network analysis, a “path” is a finite sequence of graph edges which connect a sequence of nodes and the network “distance” between two nodes is defined as “the shortest path length” (Wasserman and Faust 1994). Studies on average path length of a network reveal the efficiency of mass transport, information dissemination, or transitive communication on a network. A famous theory of “six-degree separation” describes a chain of a-friend-of-a-friend relations in real world and any two people can be connected within a maximum of six steps (Guare 1990; Watts and Strogatz 1998). A shorter degree of separation has been found in Twitter in which the average path length is 4.12 (Kwak et al. 2010).

In the STN framework, there exist three types of “distance”: (1) time delay (DISTtime); (2) physical movement distance (DISTspace); (3) path length based on connections on a virtual network (DISTnetwork). By analogy to the “graph-based shortest path length”, we introduce a spatio-temporal-network distance (STND) which is defined as the “shortest spatio-temporal-network path length” between two nodes with space-time constraints on a STN. Mathematically, it could be expressed as a generic equation below. Furthermore, we can calculate the average spatio-temporal network efficiency (STNE) derived from the STND of all node pairs as a global measure for quantifying the overall efficiency of information spreading on a time-varying network. The formulation for STND and STNE metrics is as follows.

$$STND = \alpha * DIST_{time} + \beta * DIST_{space} + \gamma * DIST_{network}$$
(4.3)
$$STNE = \frac{1}{{n\left( {n - 1} \right)}} * \sum\nolimits_{i,j}^{n} {\frac{1}{STND(i,j)}}$$
(4.4)

where \(\alpha\), \(\beta\), \(\gamma\) are scale factors for unifying different scales of distance measurements; in application scenarios, one may adjust three scale factors to represent the complicated interactions among three different distances. And n represents the total number of nodes in a STN and STND (i, j) is the spatio-temporal-network distance between two nodes i and j.

  • Spatio-temporal-Network Centrality Measures (STNC)

Traditionally, centrality measures have been employed in determining the relative importance of a central node within a network (Freeman 1977). These measures originate from structural sociology and have been widely applied in studying complex networks, urban structures, and traffic flow (Borgatti 2005; Crucitti et al. 2006; Gao et al. 2013b). Betweenness is a popular type of centrality measures (Freeman et al 1991; Newman 2005) to explore how node importance results from an interaction between the position and the characteristics of the network flow process. However, the classic betweenness measures don’t consider the dynamic changes of connections within time-varying networks. In the STN framework, we propose a spatio-temporal-network betweenness centrality (STNBC) measure which can be defined as:

$$STNBC_{i,t} = \sum\limits_{j = 1,\,k = 1,\,i \ne j \ne k}^{n} {\frac{{N_{j,k} \left( {i,t} \right)}}{{N_{j,k} \left( t \right)}}}$$
(4.5)

where STNBC(i, t) represents the spatio-temporal-network betweenness centrality for a node i at a temporal snapshot t (Tmin ≤ t ≤ Tmax ). N j,k (i, t) denotes the number of shortest spatiotemporal paths between nodes j and k through node i for a time period [T min , T max ], and N j,k (t) is the total number of shortest paths between nodes j and k for the same time period [T min , T max ]. As mentioned above, those shortest paths which rely on the STND measure may be different because of human movements in physical space or changes of network connections in virtual space over time. It implies that the influence which a node has over the spread of information through the spatiotemporal network may be varying.

4.5 Conclusions

There is a growing recognition of the importance of spatial and temporal dynamic relationships in explaining processes relevant to human behaviors, public health, and social activities (Tsou and Leitner 2013; Ye and He 2016; Ye and Lee 2016). The proposed STN framework can facilitate the understanding of patterns, relationships, and changes in human movements, activities, and communication in the physical-virtual space. This research aims to enable the convergence of new developments in GIS, the art of visualization, social network analytics, and social behavioral research, and facilitate the transformation of social and behavioral science research to computational modeling and analytical applications. It warrants notice that network structures, spatiotemporal and thematic properties, are essential for information spreading. One key contribution of this paper is a novel conceptual framework for integrating space-time visualization and social network analysis for the interaction between physical space and virtual space. To be better prepared for both natural and human-made crises, this framework may be used to facilitate quick spreading of official alerts and warning notifications via multiple virtual-space platforms and real-world channels. We hope that the vision introduced in this article could stimulate interdisciplinary thinking on research challenges and applications which involve the spatial, temporal, and social aspects simultaneously. Those identified research questions and challenges may draw researchers’ attention in future work.