Keywords

1 Introduction

Users’ day-to-day life is the primarily focus of studies dealing with pervasive location-based mobile systems. These studies are often classified as in “the wild” [4, 7], and are challenging for both new and expert researchers, due to the amount of time that needs to be spent in the field, to both collect data and to engage with participants. In this paper, we present insights and lessons learned from designing, developing, deploying, and conducting 3 studies with two pervasive systems - a location based social-network (LBSN) called GeoMoments, and a location-based (LBG) game called CityConqueror.

2 Systems and Studies

GeoMoments has been developed to help us explore the locative mobile social media phenomenon and view it through a social navigation and exploration lens. This is intended to provide us with a better understanding of vits influences and the ways in which we experience our surroundings. While there are several location-based social networks out there that let you access and look at the geo-tagged information they are not concerned about exploration, but rather about access to content – photographs, audio etc. In GeoMoments we deliberately restrict access to this content and enable the users to see only what is around 500m from their physical location. This is intended, and is used a feature that can enable us to further understand the ways in location-based social networks influence the creation and consumption of geographically based social information. See Figs. 1 through 4 screenshots of the various functions of GeoMoments.

Fig. 1.
figure 1

Users can see and geo-tag pictorial and textual moments 500 m around them. The georestriction is indicated by the blue circle around them (left). The moments can be viewed in a map (left) or on a separate screen via a mobile device (center) or browser (right).

Full size image

CityConqueror was inspired by the board game ‘Risk’, in which a player conquers countries on a world map, deploys units to defend his countries and attacks countries owned by other players. In CityConqueror, players can conquer territories in their physical location, deploy units to defend their territories and attack territories of other players that lie in their physical proximity. When conquering a territory, the player can give it a name that is visible to other players, deploy units to defend the territory and hide a treasure in it. Territories are conquered and plotted on a map of the “real” urban terrain, showing the player’s location. The map is covered by the “Fog of War”, a mechanism used in other popular games dealing with territorial conquests (see right of Fig. 3).

Fig. 2.
figure 2

Users can edit or delete their moments even if they are not physically near them (left). In addition, users can leave comment on other users’ moments (center and right).

Full size image

A player can uncover the Fog of War to reveal more of the map by physically exploring the urban space. When a player has physically visited a space and thereby uncovered the Fog of War overlaying it, they are able to see enemy territories in this area, even after leaving it. To drive the exploration of the map and thereby the player’s actual surroundings, the player can see glows (hot-spots) in undiscovered spaces, indicating the location of enemy territories in the Fog of War. Territories generate resources over time which can be collected to buy units to defend their own territories or attack enemy territories. To conquer a new territory, players must first defeat all enemy territories in their range. If a player attacks an enemy territory and wins the fight, they have the chance to find the treasure hidden in the enemy territory. Searching for a treasure is a mini-game within the CityConqueror (see Fig. 4).

Fig. 3.
figure 3

Territories are plotted on the map (left). A player can conquer a territory in his location (center). Territories in the Fog of War are indicated with glows (right).

Full size image

The player is given a compass that points in the direction of the treasure, the distance in meters to the treasure from the current location and three minutes’ time to find the treasure. To find the treasure the player must move to the pointed location. Furthermore, a player can complete achievements that reward actions related to exploring their surroundings and success in the game such as conquering a certain number of territories, conquering territories with a large distance between them, defending territories or attacking others. The objectives of CityConqueror is to claim as many territories as possible to generate income and consequently to be able to defend territories against attacks from others and attack others to conquer their territories. Thus, the game experience of one player is highly dependent on the actions and interactions of other players. In that way, we have implemented a salient social aspect in the game. See Fig. 2 for screenshots of the various functions of CityConqueror.

With GeoMoments and CityConquyeror we conducted three studies.

Studies 1 and 2 with GeoMoments too place over a 10-week period. The first study took place over a six-week period [2, 3, 12, 13, 17]. It aimed to explore the everyday usage of LBSN and involved 42 individuals with an average age of 31 years. The second study run over four weeks, and aimed at exploring the perception of self, space, place and LBSN usage. This second study involved 30 individuals with an average age of 28. Across the two studies the participants generated 985 textual and pictorial moments; 760 being pictorial moments and 225 being textual moments. In both studies, we interviewed the participants during the study, as well as conducting post-experience interviews.

Fig. 4.
figure 4

To conquer a territory the player must first defeat all enemy territories in the range of the territory (the pink ring in the left screenshot). Over time, a territory generates resources (center). After winning a fight the player can try to find the treasure by getting to the indicated position within 3 min (right).

Full size image

For study 3, we conducted a study with CityConqueror aimed at exploring mobility and territoriality in LBGs [9, 14, 15]. 12 participants were randomly assigned to one of two teams in order to play CityConqueror over two weeks. The teams competed. The game had a team score mechanism, which was based on the resources collected and treasures found by the players. CityConqueror was played actively and had a high participant engagement level. After the subsequent testing and team play phases, each participant was interviewed to evaluate their experience of the game.

Based on our experiences with these studies, we present and discuss insights and lessons learned based on key challenges that we have successfully overcome with respect to the design, development, execution of the study, and data analysis. The goal of this work is to give researchers and designers a practical understanding of conducting studies within the LBSN and LBG domains, and to help and support practitioners to reflect on their own approaches and possibly apply the proposed solutions.

3 Related Work

Maps and play on maps are relatively new within the study of games. Playing on maps began after the Battle of Jena-Auerstedt wherein the Prussian military sought to use the techniques Napoleon had used against them (topographical mapping, disconnected command structures). The answer to this was a game called Kriegspiel [10]. The appearance of this game pushed not only military training, but the seeking of information hastened the formation of tools, mapping techniques, and ways to compute that information. After World War 2, this way of using maps as training were monetized and sold as self-contained games called war games. One of the games that embraced the features of war games was Dungeons and Dragons (D&D), a role-playing game wherein players take on the role of heroes and explore maps cooperatively as a party. D&D formed the basis of playing games with and on maps and this style of play was digitized as the first computers were made available to enthusiasts [10]. Since that time, role-playing games like Pokemon have embraced the style of play that sees players exploring maps [19, 20]. LBGs are a new iteration of this style of game and formed the basis of our design [18].

There are several challenges when one embarks in studying mobile technologies in ‘the wild’. Even though a significant number of studies have been conducted in the domain, only a handful of researchers have documented the key challenges and how they overcame them. For example, Barakovic and Skorin-Kapov [1] discussed extensively the importance of a well-planned study, and described as the most challenging aspects: i) the specification of methodology, ii) identification of dependent and independent variables, iii) choosing the user test subjects, iv) determining the testing scenarios and the environment; v) and the rating scales. Similarly, Earl [11] when examining interaction patterns of smartphone users ‘in the wild’, identified and documented that volatile file systems (such as un-mounting/mounting the file-system leads to data loss), energy constraints, third-party applications, nonlinear time, and malicious users were the main issues in his explorations.

Ferreira et al. [8] expanded upon these and discussed an additional set of challenges and considerations that need to be taken into account. These include: i) careful planning and evaluation of time and effort for the deployment of a research tool, ii) the number of participants, iii) non-biased participants (age/gender), iv) the amount of collected data, v) the representativeness of the data, the vi) need of conducting the study non-intrusively, and vii) a need of an introduction of a pilot/testing study before every deployment or study as there is a need for the users to become familiar with the application. This work aims to build upon and expand this body of work by presenting a set of key challenges we encountered during our studies with GeoMoments and CityConqueror, and how we overcame these challenges.

4 Methodology

Our approach for collecting the lessons learned involved a three-step process and was similar to [5, 6, 16]. Firstly, we collectively divided the analysis of these studies in four phases: design, development, and execution of the study and data analysis. Each author proposed a set of challenges that s/he encountered in each phase and felt that it was important. Secondly, we clustered the various proposals of the most important challenges that each author had independently prioritized relating to her/his own study. Thirdly agreed upon a common focus of the employed solutions and guidelines.

5 Findings

The understanding of smartphone users’ experience is becoming vital in developing new mobile interaction concepts for LBSN and LBG. The successful approach for these experiments requires rigorous design, development, execution, and analysis of the collected results, in order for the important variables and their correlations and causalities to be captured, analyzed and understood. In this section, we draw upon our studies with GeoMoments and CityConqueror and aim to provide practical lessons learned on how to conduct user studies with LBSNs and LBGs. Our goal is to provide an overview on a set of challenges that we overcame in our studies, and to explain the solutions and guidelines we employed to the most critical challenges we identified.

5.1 Design Phase

Ethics and Consent Approval. This is one of the first considerations when designing a study. Depending the country and the institution where the study is conducted the rules and regulations may vary. Therefore, it is important to dedicate enough time for the approval as committees may ask to change the study method, hence, delaying the study, because of its late refactoring.

Pre-selection Process. The pre-selection process (or enrolment survey) should be open enough to involve as many participants as possible, designed in a way that explores the domain, and asks/discuss probable situations that the potential participants encounter daily.

Defining the Variables. This phase is very important and must be treated with caution very early in the design process because i) it defines the data and interpretations of the data throughout the study, ii) influences the ethics consent, iii) the procedure on how these variables are collected and IV) by consequence the outcome of the study.

Rewarding Participants. It is advised that the study participants receive an award if their participation requires a direct interaction (e.g. answering multiple questions in a day). In our studies the payment was a flat rate that got compounded depending how many times they would come back to answer questions throughout the study.

5.2 Development Phase

Device Issues. GeoMoments, and CityConqueror has been developed for Android OS. If issues with devices emerge during the study, they must be evaluated if a device specific solution is worthwhile, or exclude the device’s users from participation in our study.

Collecting Smartphone Data. The frequency of data collection on the smartphone and the method to be used need to be chosen very carefully because they can influence the mobile application’s performance, battery life, and data storage. As such only necessary data must be collected without jeopardizing the normal use of the participant’s phone.

Testing. It is important to test the application in the lab as early as possible to find any technical issues that may compromise the study. This phase may span over several weeks before the lunch of the study depending on the complexity of the LBSN. We suggest that internal testers conduct scenario testing with different devices, and stress test the backend extensively.

Recruiting Participants. As GeoMoments and CityConqueror was developed for Android OS it was challenging to find enough participants ready to commit for 4 to 6 weeks of studies. This “recruiting” stage lasted several weeks and we had a lot of back and forth with several participants. The solution was to post advertisements in community forums, downplay the required continuous interaction with us throughout the study, and offer monetary compensation depending the number of interactions with us.

5.3 Execution Phase

Visualization of the Data. For the purpose of our study that it was important to show the participants the data we gathered from them through GeoMoments and CityConqueror during our weekly interactions with them. We experimented with a range of different visualizations and after a lot of ‘trial and error’ we created the following guidelines that we followed throughout our project: I) Avoid representation of geocoded traces and posts quantitatively as most participants are not able to understand them and they do not help in our discussions with them, II) Show only data related to the questions at hand. Don’t show all their data to the participants at once as too much information will bias their overall behavior during the study and may change routines and related activities based on the results presented.

Participant Motivation. The participants must be kept motivated to fulfill the requirements of the study and provide unbiased data results. Motivation in our project came from monetary compensation, and a sense we conveyed to the participants that they were part of the project and helped us answer important research questions.

Privacy. Even though general rules should be described in the ethics consent there are a number of practical issues that should be taken into account. These include: i) if the study requires connecting participant’ smartphone to a PC, disable any automatic download of personal information, ii) no overlapping interview schedule or involving third individuals outside of the study, not disclose other participants’ opinion and experience to other participants (this includes not referring to other participants when giving examples during interviews), iii) offer the possibility to switch data logging off for a limited time when participants want extra privacy,

Performance Issues. Participants’ phone performance issues may influence the final results of the study and the motivation of the participant. As such any complains regarding phone performance issues have to be investigated and addressed immediately.

Software Bugs and Related Behavior. In our studies we encountered a wide variety of Android OS devices that each created a wide array of bugs and weird behaviors. These ranged from high/low sampling frequency from the GPS, disturbing vibration, inconsistent logging etc. Our approach was to investigate and see if the bug can be easily, identified, replicated and solved fast while at the same time making sure that our solution did not create even more problems in the future.

5.4 Data Analysis Phase

Data Synchronization and Format. The format of the collected data should be well designed, so that post processing the data does not take too much extra effort. It is suggested that collation and synchronization of the data from various sources/databases etc. is an ongoing process through the study.

Analysis. Methodological issues and approaches aside, the data analysis process should be kept in digital format and kept separated per participant to lead to a faster data analysis. Themes, trends and directions as emerge from the data must be derived by at least 2 different coders and should be able to explain a particular set of situations when discussing and leveraging the study results.

Closing the Project. Closing the project involves a significant amount of challenging work. Extra care should be taken when addressing any finance issues, writing the final project reports, etc. However, admin work aside we suggest that one should confirm that the project has met all research objectives, and requirements from sponsor, customer, and stakeholder, verified that all deliverables have been delivered and accepted, and that the exit criteria have been met.

6 Conclusion

In this paper, we have touched upon a number of issues one may encounter and potential solutions one may employ when designing, developing, and deploying LBSN and LBG. The lessons learned relate to the lifecycles of the studies, and illustrate that when one should be prepared and flexible enough to accommodate any issues as they arise in a professional manner considering not only the results of the study but also the participants and the researchers involved. The goal of this work is not to provide specific solutions to any kind of challenge the researchers or practitioners may encounter, but to provide to researchers a practical view on conducting studies within the LBSN and LBG domains, and to help and support practitioners to reflect on their own approaches. As such future work should have a more formal approach. This is proposed to involve a systematic review of past projects from multiple authors, and focus on the development of a theoretical framework for conducting human subject studies in the wild.