Keywords

1 Introduction

Lifelogging is the act of digitally recording aspects and personal experiences of someone’s life. Some people are interested in logging their life’s activities for fun, medical purposes or diary applications [1]. It is important for many individuals to retrieve moments and events such as trips, weddings, concerts, etc. Reminiscing previous events among a group of people not only helps in remembering those events, but it also creates tighter social bonds and improves relationships among them [2]. Aiding memory is also one of the benefits that people gain by logging their life. For example, a lifelogging system can be used as an external memory aid that supports a person with memory problems by using reminiscence therapy [3].

A natural way to structure lifelogs is in the form of activities; for example having lunch, sitting in the park, shopping, attending a seminar, etc. This structuring requires techniques for reasoning and inferring of activities from the logged data. The logged data is part of the lifelogs and the granularity, as well as the types of data, can vary. However, the basic context should be captured to infer activities. This basic context have been analysed and identified as identity, location, activity and time, where locations and activities are of special importance [3, 4]. Context data could be captured by mobile devices carried by the user such as wearable sensors. It is good, however, to use a single mobile device when logging, as the number of devices the user needs to carry should be kept to a minimum.

Just structuring data into activities based on context may not be sufficient for efficient retrieval and to support people reviewing their life experiences. Both context (e.g. time, locations and places) and content (e.g. images) need to be aggregated and segmented into the activities and be given semantic meaning. In previous work the authors have explored using known places to create this semantic meaning [3]. However, this approach is limited to predefined places. A desired solution would be finding places of importance and then inferring activities automatically. In this paper the authors introduce an approach to detect new places and then infer activities automatically relying solely on time-stamped location data. Location and time are rich and easily accessible sources of context information that are relevant to find places of importance, where the user spent significant time. Being for a period of time in a significant place might be an indication of some activities happened in the place. The first problem that the paper addresses is: “How can places of importance be recognized and activities be inferred based on location data and time?”

Once lifelogs are segmented into activities, they can be annotated with content, such as images and descriptions. Images play a vital role in enriching the logs and in supporting reminiscence processes in a lifelogging system [5]. Images can be captured automatically by purpose-built devices (e.g. SenseCam which is further described in Sect. 5) or by a smart-phone carried in a way that allows it to capture images. However, the information and the images still need to be presented to the user in a way that takes advantage of the structured lifelogs. The second problem that this paper addresses is: “How can structured lifelogs be presented so the user can review and retrieve the life experiences?”

The rest of this paper presents the work done to address the problems listed in the introduction and is organized as follows: Sect. 2 shows what algorithm has been used in this work to recognize new places. The calibration of the chosen place recognition algorithms is presented in Sect. 3. Section 4 discusses the algorithm that has been used to infer activities. The development and deployment of the prototype application, which organizes the logs and presents them to the user, is the topic of Sect. 5. Section 6 presents the user evaluation of the proposed system. Section 7 discusses the research questions and Sect. 8 presents some of the related work. Finally, Sect. 9 concludes the paper and presents the future work.

2 Recognizing Places of Importance

Recognizing places is important because the places people visit contain hints towards the activities taking place. One of the common approaches for discovering interesting patterns and data distributions from location data is density-based clustering algorithms [6]. These algorithms can infer information of areas where the user spent significant time when having location data logged by a mobile sensor carried by the user [7, 8]. The algorithm that was used by the authors in this work is Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [6].

The proposed Place Recognition Algorithm relies on GPS points as a source of location data. The adopted approach is depicted in Fig. 1.

Fig. 1.
figure 1

New places recognition – action flow

Full size image

The aim of the clustering algorithm is to identify places of importance to the user, which are previously unknown in the system. These places can be confirmed and labelled by the user while reviewing the lifelogs. If the user confirms a place, the system will add the coordinates that correspond to this place and define the place as a known one. The algorithm compares each GPS point with all previously known places. If the point belongs to a known place, the algorithm will remove it from the input set, but keep it for inferring activities later on. If the point does not belong to a known place, the algorithm will keep it in the input set for clustering. The GPS points in the input set are then clustered and aggregated regardless of time. Such clusters are signs of places where the user spent significant time. After the clusters are identified, the system constructs the convex hulls to estimate the geographical boundaries of the places [9]. Figure 2 illustrates the view of the place clusters after implementing DBSCAN over the location data and constructing the convex hull.

Fig. 2.
figure 2

Recognized places

Full size image

3 Calibrating the Place Recognition Algorithm

DBSCAN algorithm uses two parameters: the Radius, the range around a point where other points in that range are considered neighbours, and MinPts, minimum number of neighbours that a point needs in order to not be declared as noise. After setting the parameters, the algorithm forms clusters using the density of local neighbourhoods of points. This approach is repeated for all the points resulting in many clusters with different arbitrary shapes. All points that do not belong to any cluster are considered noise. A suitable set of parameters is the set that results in fewer numbers of place recognition errors. In order to evaluate the number of place recognition errors that correspond to different parameter sets, real-life data were collected and labelled manually, the possible error types were defined and then the performance of different parameter sets was estimated with respect to the identified error types.

A Windows Mobile application has been developed to log GPS tracks periodically every 30 s. When connecting the logging device to a computer, the application transfers the logs as an XML file that contains longitude, latitude, logging time, speed, and number of satellites. Three users have done the data collection over a period of six months. The users were asked to carry a mobile device, with the application installed, during the day. By the end of the day, the user connects the mobile to a computer to transfer the logs.

25 randomly chosen logs were analyzed to determine the best parameter values of the DBSCAN algorithm. Each log has data collected during one day. Logs were manually analyzed and essential places were identified based on observation. The manual analysis of the collected data revealed that there are 4 types of possible errors:

  1. 1.

    The algorithm detected a cluster that does not correspond to any real-life place.

  2. 2.

    The algorithm merged two places into one.

  3. 3.

    The algorithm separates one place into two different ones.

  4. 4.

    The algorithm did not detect an essential place.

The DBSCAN algorithm has been implemented using JavaScript and the results have been shown through a web application and manually processed to identify errors of different types. The application shows a map with all collected points during the day on the left side, and the clustering results after applying DBSCAN based on the Radius and MinPts on the right side. Figure 3 presents part of the results when running DBSCAN on one selected log with 20 m as a Radius and 3 points as MinPts. The points that are marked by 1 belong to one cluster while the points that are marked by 2 belong to another cluster.

Fig. 3.
figure 3

DBSCAN implemented in a web application

Full size image

Different reasonable values of the Radius and MinPts were tested to find out what errors they produce. For each log, the following parameters sets were considered: every possible MinPts from 2 to 20 with the step of 1, combined with every possible Radius from 5 m to 60 m with the step of 5. The aim is to find the minimum values of MinPts and the Radius that result in fewer numbers of errors for each log. The priority is to reduce the errors of type 4 when the algorithm does not detect essential places. The number of errors of this kind tends to grow with decreasing the Radius or with increasing MinPts. After determining the best values for each log, the average of those values is calculated to find out a representative value. Calculating the average value for the 25 logs gave a 3.28 for MinPts and 39.2 for the Radius. Since increasing MinPts might result in increasing the number of undetected places, the value 3.28 is rounded to 3. Thus the parameter values that yield the best results for the DBSCAN algorithm are: 3 for MinPts and 39.2 m for the Radius.

4 Inferring Activities

Once places of importance have been identified then they can be used to infer activities. The following properties of an activity are identified:

  1. 1.

    An activity occurs at a place.

  2. 2.

    An activity takes a certain amount of time.

Based on these properties, a set of GPS points is an indication of an activity if:

  • The points belong to the same place.

  • The points are sequential in time.

The main idea of the activity inference algorithm is to decompose all place clusters into sub-clusters that do not overlap with each other in time. Overlapping occurs when the user leaves place A, for example, to place B then comes back later to place A. The fact that GPS points are naturally ordered by time makes calculating the timeframe easy. Clusters that correspond to the visited places are added to the input set of the activity inference algorithm. Figure 4 illustrates the view of activities after decomposing Place 1 (which is shown in Fig. 2) to sub-clusters that represent activities.

Fig. 4.
figure 4

Recognized activities within a place

Full size image

The user can label the activities when reviewing the logs. For instance, having meeting in the office, or working on the computer in the office. Those are 2 different activities that can be labeled differently even though they happened in the same place.

5 Implementation and Deployment

For lifelogs to be useful they need to be structured and presented to the user in a way that will give a good overview of content and data. This section presents a prototype that consists of mobile devices for capturing images and contexts, and an application for reviewing the gathered data. The mobile device for capturing images that is being used in this prototype is called SenseCam. SenseCam is a wearable digital camera, which keeps a digital record of the activities that the person experiences [10, 11]. Figure 5 shows the main interface of the application after transferring the logs of one day. This interface consists of 2 columns, where one column presents places and the other one presents activities.

Fig. 5.
figure 5

The main interface of the lifelogging application

Full size image

When connecting the two devices, the SenseCam and the smartphone, to a computer with the prototype application installed, the system performs the following steps:

  1. 1.

    Transferring the logs in the form of XML. The logs consist of time-stamped GPS data and time-stamped images.

  2. 2.

    Analyzing the GPS data to identify periods of time where the user visited known places during the day.

  3. 3.

    GPS points that do not correspond to any of the known places are aggregated, using the DBSCAN algorithm, into clusters that represent new places. The Radius is set to 39.2 m and MinPts is set to 3.

  4. 4.

    Inferring activities based on the places using the method presented in Sect. 4.

  5. 5.

    Associating SenseCam images with the recognized places and the inferred activities based on time.

  6. 6.

    Showing the results on the main interface in a chronological order.

Figure 6 above shows the place page when reviewing. When reviewing a place, the system shows the constructed convex hull from the GPS points that correspond to the place. In addition, SenseCam images that have been captured when the user was in the place are shown. The user can choose a representative image for the place using the available SenseCam images. If the user confirms the place, the system will save the chosen image as the representative one together with the coordinates that correspond to this place. Thus the place will be known and detected automatically by the system if the user visits it again. This will improve the system’s knowledge of important places, which will increase the level of automation in detecting the user’s movements. When reviewing an activity, the system presents all SenseCam images that have been captured during that activity. The system also shows all the GPS points that correspond in time to this activity on a map. It is possible for the user to choose certain images to associate with the activity among the whole set of images. Figure 7 above shows the activity page when reviewing.

Fig. 6.
figure 6

Reviewing a place within the lifelogging application

Full size image
Fig. 7.
figure 7

Reviewing an activity within the lifelogging application

Full size image

6 User Evaluation

6.1 Participants

To evaluate the implemented lifelogging application, the authors asked 12 users to carry the equipment for 1 day. 6 males and 6 females took part in the evaluation and their ages ranged from 22 to 43, with a total average age of 28.2. Each participant carried the mobile device (to collect GPS data), and wore the SenseCam around the neck (to capture images automatically). The data was then transferred to a stationary computer and presented through the lifelogging application to the participants. The users were asked to pick a day where they have a plan to move around and do different activities. Each participant reflected the experience of using the system by answering a survey.

6.2 Results

The survey contains 13 questions. The questions were designed to give an overview of the participants’ satisfaction of the application. This includes the place recognition results, the activity inference results, the presentation of the results on a map, and having places and activities annotated with SenceCam images. For each question, the average value that represents the answers of the 12 participants was calculated to give an overview of the results. The participants were first asked about how many places they were expecting to see as significant ones based on their day. The number of the expected places ranged from 2 to 6 and the average value of the expected places for all 12 users was 3.7. After processing the data through the lifelogging application, the participants looked at the place recognition results and see how many significant places were recognized correctly. The average value of the correctly recognized places for all 12 users was 3.5. This means that the application could recognize significant places with an accuracy of 94 % (3.5 out of 3.7).

In very few cases, the application recognized several important places as one, or recognized something irrelevant. However, the participants considered this point as a minor one. In addition, the application never recognized one important place as several ones for any of the participants.

All the questions that reflect the participants’ satisfaction were ranked from 1 to 10, where higher value indicates better result. Showing the GPS data on a map was helpful in recognizing the place by the participants with an average satisfaction value of 9.4 out of 10. The users gave a positive opinion as well regarding associating SenseCam images to the recognized places with an average value of 9. The overall satisfaction with the place recognition results was 8.9 out of 10.

Regarding the inferred activities, the participants were first asked if the activities made sense to them, and the average satisfaction result was 8.5 out of 10. Associating SenseCam images to activities was more helpful than associating them to places with an average value of 9.5. However, the overall satisfaction of the activity inference results was less than the satisfaction of the place recognition results with an average value of 8.3 out of 10.

Participants indicated that it is hard to recall activities without looking at the SenseCam images with an average value of 7.9. All participants found it very helpful to have auto-captured images to recall all the details of the day. Finally, the overall satisfaction of the whole lifelogging application was 8.8 out of 10.

6.3 Participants’ Comments

The overall reaction of all participants was very positive with minor concerns. One participant wrote: “I think it is a very useful device to have/carry with you on a daily basis. I see this device and program to be suited for people/jobs with high level of diversity and unknown situations in their daily life. A very good use of this memory program would be for policemen, security guards or hospital healthcare”

Some users found the application as a cool tool to use. They think that the concept is very interesting, and that having a lifelogging application is a fun thing “I thought that this is a very cool technology, I was very excited to look back at the data”. Another quote: “it is interesting to see how much of my life I can remember. There are some details that I couldn’t tell before reviewing my day by this application. This is cool!”

Some participants think that this application can be a memory aid to recall previous experiences “Having your days structured as activities is very helpful to recall your past. It is very hard for me to remember what I did 2 weeks ago, but not with this application. I can even recall any day in details when I look at it as a sequence of images. I see a big value of using this technology if the person has memory problems”. Another participant said: “I think this technology can be helpful for both short-term and log-term recalling. For short-term, remember where I put my keys by looking on the images. For long-term, review what I did in the past on a specific day. However, the application is more interesting for outdoor activities, and I would prefer to use it if the camera is integrated in my cloth”.

Some users found the application helpful but not for everyday life. For instance, one of the participants commented: “I wouldn’t use this application on daily basis. However, it will be great to have it when attending important events to keep track and memory of that event”. Another participant said: “yes, I like this but more on special occasions rather than every single day. Maybe when I’m on holiday or at a conference and I meet lots of new people”.

7 Discussion

This section discusses the results of the efforts with respect to the research questions.

The first addressed question is: “How can places of importance be recognized and activities be inferred based on location data and time?”

Places can be recognized relying solely on time-stamped location data using the DBSCAN algorithm. DBSCAN aggregates GPS points into clusters based on the density of points. The authors calibrated the density-based algorithm based on data collected by three users over a period of six months. The best parameter values for DBSCAN that result in fewer numbers of place recognition errors are 39.2 m for the Radius and 3 points for MinPts. The DBSCAN algorithm results in clusters that represent places visited by the user. After the clusters are identified, the system constructs the convex hull to estimate the geographical boundaries of the recognized places.

Activities are inferred based on the known places and the essential places that are defined in the previous step. An activity is represented by a set of GPS points which belong to the same place and which are sequential in time. The system searches within the defined place clusters and splits them into sub-clusters that do not overlap in time. Each sub-cluster represents an activity that happened in a certain place at a certain time. The timeframe of each activity is the time between the earliest point and the latest one within the sub-cluster. A cluster, which represents a place, might be divided into several sub-clusters, which represent activities happened in the same place at different time.

The second addressed question is: “How can structured lifelogs be presented so the user can review and retrieve the life experiences?”

The lifelogs, which are structured based on places and activities, are presented through a prototype application that answers the following questions:

  • When did the activity take place? The timeframe of the activity is presented based on the identified corresponding time-stamped GPS points.

  • Where was the activity? The place where the activity happened is presented on the map based on a convex hull of the corresponding GPS points.

  • What was the user doing? The presentation of the activity is based on the auto-captured images, which were taken at the time of the activity.

A SenseCam can be used to capture images automatically while a mobile device can collect GPS points during the day of the person. The system transfers all the logs when connecting those portable devices to a computer, and then defines places and activities based on the GPS data. SenseCam images are then associated with those places and activities based on time and presented to the user for reviewing and adjustment. Adding SenseCam images, as content, to the clustering results helps the user in naming places and activities when reviewing.

If the user confirms a cluster as a place, the coordinates that correspond to this place are saved and the place will be known and detected automatically next visit. Therefore the system can improve its knowledge about the user’s preferable places. Saving activities will just save the data and make it available for later retrieval. The system thus presents the structured lifelogs as places and activities associated with SenseCam images. The system helps the user to retrieve or share previous moments in life based on places or time. For example, the user can review all the activities that happened in a certain place, such as the university, or at a certain time, such as the New Year Eve.

The system was evaluated by 12 participants who used the equipment for 1 day. The overall satisfaction was positive with slightly better results for the place recognition results (8.9 out of 10) than the activity inference results (8.3 out of 10). The overall satisfaction of the whole system is 8.8 out of 10, meaning that this lifelogging system could be accepted and integrated in the daily life. Combing images with the aggregation results, together with presenting the GPS data on a map, helped the participants in reflecting and recalling their life experiences easily.

8 Related Work

The DBSCAN algorithm was used by the authors when clustering the location data. Another common clustering approach is K-means [12]. Ashbrook et al. [13] used a variant of the k–means clustering algorithm that used GPS data in defining locations of the user. All the points within a pre-defined Radius are marked, and the mean of these points is computed. The system will do calculation again based on the mean and the given Radius to find a new mean value. When the mean value is not changing any more then all points within its Radius are placed in a cluster and removed from consideration. This approach is repeated until there are no more GPS points left. The main limitation of this approach is that the Radius should be set in advance and then the clustering algorithm will rely on that value. Density-based algorithms overcome the limitations of the K-means clustering method [6]. The advantages of using DBSCAN over K-means are mentioned by Zhou et al. [14]: DBSCAN is less sensitive to noise, it allows clusters of arbitrary shape, and it provides deterministic results.

Palma et al. [7] used a variation of the DBSCAN algorithm to discover interesting places in trajectories, which are previously unknown. Trajectories are observed as a set of stops and moves, where stops are more important. The authors in [7] calculated the distance between points along the trajectory instead of using Euclidean distance, and they used minimal time instead of using minimal number of points MinPts, for a region to be dense. The absolute distance (Eps) is used to calculate the neighbourhood of a point. The choice of Eps requires knowing the percentage of location points, which corresponds to stops.

Andrienko et al. [8] defined the trajectory of an object as temporally ordered position records of that object. The authors in [8] looked at the characteristics of the movement such as instant speed, direction, travelled distance, displacement and the temporal distances to the starts and ends of the trajectories. These characteristics are then represented as dynamic attributes that can give an indication of movement events. For instance, having low speed in some areas can be an indication of specific events belonging to those areas. The events are clustered according to their positions and time, and then used to extract places. Repeated occurrences of events in places are calculated by means of density-based clustering, and those places are defined as interesting ones to the user. The result was defining places of interests from mobility data by analysing place-related patterns of events and movements. However, the work presented in [8] relied on the data collected by many users in the area, while our work is designed for detecting and logging personal preferences, so activities in our work represent the personal life experiences of the user. In addition, no prototype application was done in [8] so the user cannot review and save the detected places and events for later retrieval.

The effect of using locations and images on memory recall has been tested by Kalnikaite et al. [15]. In their work, SenseCam images are associated with locations based on time and then presented to the user through an application. However, images are associated without the use of any particular clustering techniques. Another application that presents groups of images on a map based on their locations has been created by Toyama et al. [16]. All the images are tagged by the location data and stored in a database, and then the application groups the images and shows them on the map based on the tagged location. This application lacks the automatic detection of important places as it relies mostly on the tagged data of the images.

9 Conclusions and Future Work

This paper presented an approach that relies on location data and images to organize the lifelogs of someone’s life. Location data provides a context source that can be used to recognize places and infer activities. Images, as content data, can be then associated with those recognized places and inferred activities, and be presented to the user for reviewing and adjustment. The introduced prototype system structures and presents lifelogs based on places, activities and images that can be available for later retrieval. The system therefore provides a digital tool for people to reminisce and share their life.

The next stage of our work is improving the inference of activities within the lifelogging system using the same set of devices. Sensor-readings in SenseCam can be used with image processing techniques to better reason about daily activities. This will also help the system distinguishing between different activities that usually happen in the same place, which will improve the activity inference task.