Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

10.1 Introduction

Traditionally, in terms of us remembering important events and activities of our own lives, the written diary has been the first choice. But with modern technology, the diary recording of our lives is now digital. Emails sent and received, web pages viewed and interacted with, photos and videos taken, viewed and shared, blog postings and tweets written, on-line social network activities, calendars, goods purchased and so on. These represent our present day. Through advances in wearable sensors, we now have the capability to automatically record at large–scale the places that we have been to, things we have seen, people we encountered and how active we are. Almost everything we do these days, on-line and physically, is in some way monitored, sensed and logged. Further, with the introduction of Facebook’s Timeline (Sittig and Zuckerberg 2010), we have come to accept, or maybe we just ignore, this massive surveillance of our lives from a variety of sensors. This can be our smartphones, e.g. Reddy et al. (2007), Qiu et al. (2012), intelligent devices like TVs and cars but also the surveillance cameras, recording of travel data at subways and buses and purchases with credit cards. The creation of such multi-modal human media archives, or lifelogs, will be commonplace.

In 2011, the European Union agency ENISAFootnote 1 evaluated the risks, threats and vulnerabilities of lifelogging applications with respect to central topics such as privacy and trust issues. In their final report, they highlight that lifelogging itself is still in its infancy but nevertheless will play an important role in the near future (Daskala et al. 2011). Therefore, they recommend further research in order to influence its evolution to “be better prepared to mitigate the risks and [to] maximize the benefits of these technologies”. Various researchers have worked on handling lifelogs, starting from identifying important events in the data stream, analyzing the sensor data to generating semantically meaningful annotations that describe these events and the implications on privacy (O’Hara et al. 2008; Gedik and Liu 2008).

Due to the characteristics of the capturing devices, we know that these lifelogs can be large and contain chronological ordered data. Captured materials may include text, photos/video, audio, location, Bluetooth logs and information from many other sensing modalities, all captured automatically by wearable sensors. Experience suggests that it can be overwhelming and impractical to manually scan through the full contents of these lifelogs. A promising approach is to apply visualization to large-scale data-driven lifelogs, as a means of abstracting and summarizing information.

Data visualization supports representation of abstract information, which facilitates further exploration (Shneiderman 1996). Previous studies mainly focused on plain numerical data, often in the form of charts, tables, figures, diagrams, etc. However, in the case of lifelogging, data generated from multiple sensor sources are more complex, requiring careful consideration of how to summarize and represent this information. In visual lifelogging, it is common for the sensing devices to automatically capture thousands of photos and tens of thousands of sensor readings per day (Kalnikaite and Whittaker 2012).

We argue that lifelog visualization should be capable of displaying the sheer quantity of mixed multimedia content in a meaningful way, taking into consideration user behaviors and needs. It requires incorporating domain knowledge into the data aggregation, compression and rendering process. Through different means of interaction, the visualizations should provide an insight to help users improve self-awareness, support retrospective memory access, and furthermore, discover their own stories and life events. Hence, lifelogging provides a new domain for the development of adaptive interactive systems.

In this chapter, we discuss the use of context-based HCI design templates, which support a more systematic approach in exploring the match between user needs and interaction provision. Template-based user interface development aims at providing a systematic approach to specify the user interface by means of models. It helps us to design for maximum usability. We will focus on different user scenarios, including (a) the creation of personal lifelogging legacies, (b) energy expenditure measurements, (c) reminiscence therapy and (d) social activity capturing. These UI templates serve as guidelines towards increasing the use of HCI design patterns for lifelogging devices.

The chapter is structured as follows. In Sect. 10.2, we provide a brief overview over existing lifelogging technology. Section 10.3 introduces use cases that demonstrate the application of these technologies, In Sect. 10.4, we introduce different interaction patterns for different lifelogging scenarios. Section 10.5 concludes this chapter.

10.2 Lifelogging Devices

Digital lifelogging is an ubiquitous concept and exists in various forms. As mentioned above, it includes any digital document one creates every day, e.g., letters that are typed on a personal computer, pictures taken with a digital camera or mobile phone, posts in the Internet, etc. Further, it can be automatically captured data such as GPS records, personal health information or ambient living awareness systems in an intelligent housing system. As the prevalence of digital logging devices, we now have the ability to gather and store large volumes of personal data under a meticulous concern of privacy. Furthermore, people have always collected mementos over lifetime. Besides, sharing digital information is already commonplace, through emails, mobile phones and social network. Focusing on an individual (i.e., the lifelogger), various lifelogging devices provide a constant stream of personal data. The functionality of lifelogging devices is as diverse as the data that is recorded by these devices. We classify these devices into four categories: (1) Portable Cameras, (2) Biometric Devices, (3) Other Portable Devices and (4) Networked Systems. All devices allow users to record some part of their daily life (GPS tracking, environmental surroundings, our body metrical information, etc.). The categories are introduced in the remainder of this section.

10.2.1 Portable Cameras

The most advanced application of personal lifelogs is the storage of pictures and videos that depict people’s life. Various devices have been introduced that can be used to achieve this task.

An early product is Microsoft’s SenseCam (Hodges et al. 2006), a camera with fisheye lens and various additional sensors which is worn around the neck. SenseCam has been used extensively in the MyLifeBits project (Gemmell et al. 2006), a research project aimed at creating lifetime storage databases. SenseCam is a small lightweight wearable camera used to passively capture photos and other sensor data (temperature, accelerometer, magnitude, infrared ray etc.). A similar product will soon be launched by the Swedish startup company Memoto,Footnote 2 who successfully raised money via crowdsourcing to start the production of a small lifelogging camera. Their funding campaign attracted world-wide media attention (e.g., by Wired Magazine, Wall Street Journal, The Guardian, The Huffington Post, BBC, Die Zeit, and others), indicating the growing attention that lifelogging receives nowadays. Other potential lifelogging cameras are Google’s Project Glass, an augmented reality head-mounted display (Olsson 2012) and Looxcie (Boland and Pereira 2011), a mobile-connected, handsfree, streaming video camera.

A collection consisting of over 10.5 million personal lifelogging images (Doherty et al. 2009) has been used for the development and testing of various algorithms to automatically categorize image content. For example, researchers studied techniques to automatically segment the stream of daily lifelogging pictures into semantically coherent events and to group them in categories such as socializing, eating, travel behaviour, environments that people experience, or movements (Doherty 2009; Byrne et al. 2008; Doherty et al. 2008; Li et al. 2013). Doherty et al. (2012) introduce a browser that allows users to view of summary of daily events instead of watching all pictures of a day.

Apart from focusing on the technical handling of this big data, various research has been performed to study the potentials of lifelogging images in medical and other settings. For example, studies have shown (e.g., Berry et al.2007) that the process of reviewing lifelogging pictures has a positive effect on memory recall, with even better effects than the more traditional personal diary. The authors report that enabling patients with limbic encephalitis to relieve their day by exploring lifelogging images has a positive impact on the patients’ confidence, stress level and their ability to cope with their impairment. Further, neuroscientists observed that viewing personal lifelogging images triggers activity in parts of the brain that is associated with normal autobiographical remembering (Berry et al. 2009). Similar results are reported by Piasek et al. (2011), Pauly-Takacs et al. (2011), who asked memory impaired patients to include lifelogging technologies in their daily lives.

10.2.2 Biometric Devices

Another popular domain where users rely on sensors to record aspects of their life is the health market. As shown in a recent survey (Zhou and Gurrin 2012), many sports enthusiasts are keen to record their sports life. There are a bundle of biometric devices that are capable of logging biometric information such as Galvanic skin response, skin temperature, heart rate or increased sweat production, sympathetic nervous activities, etc. These devices are mainly used to interpret physiological responses that provide evidence for the personal well being. Commercial products include ReadiBand from Fatigue Science,Footnote 3 a tiny device that has to be worn around the wrist and Bodymedia,Footnote 4 which is designed to determine energy expenditure and other biometric information. Targeting the amateur sports market, devices such a Fit-BitFootnote 5 and Nike+iPodFootnote 6 promise fitness monitoring facilities based on internal sensor data. Both devices have in common that they are rather small and are comfortable to wear. Given their small size, these small devices play an essential role in the evaluation of sports and health related studies (e.g. Cole et al. 2004). Consequently, the underlying technology, i.e., the analysis of sensor data such as accelerometer data to determine energy expenditure has been studied extensively, e.g., by Albinali et al. (2010), Swartz et al. (2000), Montoye et al. (1983). Further related research includes the analysis of sensor data to determine biometric features include device-independent gesture recognition (Kratz et al. 2011) and the continuous and non-intrusive user identity verification in real-time environments (Messerman et al. 2011).

10.2.3 Other Portable Devices

Apart from devices that record visual or biometric data, various other portable devices exist that can be used for logging parts of our lives. For example, GPS data trackers constantly create time-annotated location points that allow tracking movements. These devices are often used to track animals, (e.g. Weimerskirch et al. 2002; Schofield et al. 2007), vehicles, employees, children and/or elderly people. Further, GPS sensors are used to geotag pictures and record travel, hiking, cycling, flying or racing routes, thus revealing information such as individual travel behavior (Clements et al. 2010) or general places of interest (Kennedy and Naaman 2008). Rekimoto et al. (2007) rely on WiFi signals to create such patterns. Further, Miyaki and Rekimoto introduce a mobile sensor which measures ambient air environment (Miyaki and Rekimoto 2008). By plugging the sensor on a mobile phone, a detailed record of air quality will be created, thus revealing details about one’s personal ecological environment.

A promising portable device that comes with various internal sensors is the smartphone. A standard smartphone contains a wealth of sensors: a camera, a GPS chip, an audio recorder, a 3-axis accelerometer, an ambient light sensor and a digital compass. In essence, a standard smartphone contains the full specification for recording the entire physical and digital environment that a user is experiencing. Additionally, Bluetooth headsets that can record, store and upload 5+ hours of medium quality video are already available for less than EUR 200.

Hence, we have the ability to gather and store large volumes of personal data (location, photos, motion, orientation, etc.) in a very cheap manner. Exploiting the idea of using a smartphone as the primary tool for gathering lifelogging data, MIT’s Media-Lab introduced the Funf Open Sensing FrameworkFootnote 7 which creates a snapshot of the user’s life by recording data from over 30 sensors, including Wifi signals, locations, usage of apps on the phone, and others. Data will automatically be uploaded to the cloud and can then be analyzed by the user (Aharony et al. 2011). A similar framework has been presented by Rawassizadeh et al. (2011). Further, a framework is currently developed within the SenseSeer project.Footnote 8 The aim of this project is to develop a software platform that will automatically record, annotate and interpret a user’s life activities, based upon the sensor data that they gather unobtrusively as part of daily life. Using a smartphone, the system can recognize activities like actions (sitting, eating, driving), people (who is nearby), places (physical locations, indoor/outdoor, office/home) or nearby objects (screen, steering wheel, people). In addition, the SenseSeer application supports automatically taking pictures, thus emulating the functionalities of a SenseCam (Qiu et al. 2012).

10.2.4 Networked Systems

The vast majority of devices that are used to record parts of our life are used to monitor the environment (e.g. CCTV cameras, digital time sheets) or to provide some customer-oriented services (e.g. ATM machines, Online web services, etc.). Combining all personal data streams that have been recorded by these networked systems can result in a detailed description of our every day activities. Elliott et al. (2009) propose a framework for capturing all personal data in a personal archive. However, the data that is recorded using these devices is often beyond our control. Besides, these systems are often networked with other systems and services, hence distributing this personal data. Therefore, combining personal data under these conditions should be examined with caution.

Recently, the European Commission funded the Dem@CareFootnote 9 project via its FP7 funding scheme. The aim of Dem@Care is to provide multi-parametric remote monitoring and enabling for persons with dementia, allowing them to live an independent life in their community. The analysis of multi-parametric sensor data has also been studied within the context of the SmartSeniorFootnote 10 project. The aim of this project was to improve life quality of elderly people at home, e.g., by enabling them to sustain mobility, safety and independence.

10.3 Use Cases

As shown in the previous section, various devices exist that can be used to record aspects of our lives. Bell and Gemmell (2009) argue that this is just the beginning of a “total recall revolution”. The authors researched digital lifelogging for several years, concluding that technology is developed (or might be developed soon) that allows us to remember everything. In this section, we provide four use cases that illustrate the motivation that users of lifelogging technologies might have to create such digital memories.

10.3.1 Personal Lifelogging Legacy

10.3.1.1 Scenario

“John Doe is a technology enthusiast. Hence, it did not take much to convince him to buy one of those lifelogging devices that everyone was speaking about these days. Buy one of our devices, their marketing slogan said. Leave a legacy behind that really represents you: A record of your every-day activities. This promise fascinated him. Wearing a lifelogging device, he could record everything he did, day by day. People he met, places he had been to, … everything! The longer he would use the device, the more of his data would be available, leaving a “digital fingerprint” of his life. In a way, this data would keep a record of his life, the essence of his experiences. Of course it would not make him immortal, but still, with conscientious data storage, he would leave something behind that those about to follow could get back to … .”

10.3.1.2 Comment

With the rise of Facebook, Twitter and other social network services, we already have access to technologies that allow us to share certain moments of our lives online. Nadkarni and Hofmann (2012) argue that social network services satisfy two primary human needs: (1) the need to belong and (2) the need for self-presentation. Thus, we argue that above scenario is very close to real life. John could be anyone, his need to share life moments drives many users of social network services to post aspects of their lives (pictures, videos, text, …) online.

Above scenario has been addressed by Bluepatch Productions and Floating World Productions in their theatre play Oh look, hummingbirds, which premiered at the 2012 Dublin Fringe Festival. In this play, a journalist is given the experience to view the memories of a loved-one who passed away. This service is provided by a futuristic company that specializes on keeping one’s lifelogging legacy and enabling selected persons to access this data. Although the play is set in the future, the technology to record such lifelogging databases already exist and indeed, early adopters already created their own personal lifelogs covering several years of life experience (Doherty et al. 2009).

10.3.2 Energy Expenditure Measurement

10.3.2.1 Scenario

“Mandy never was a very disciplined person. Therefore, it probably is no big surprise that every diet she tried so far was meant to fail. Although she initially manages to lose some weight by changing her food consumption habits, she eventually breaks in and eats that extra piece of cake again … Following up on her medical doctor’s advise, she finally decides to get more physically active, hence losing weight by burning more calories rather than just reducing food intake. Knowing about her weak discipline, she joins a supervised fitness programme in her local gym where she is asked to monitor her physical activities and to discuss this activity log with her coach and other members of the training programme.”

10.3.2.2 Comment

As argued in Sect. 10.2.2, various biometric devices exist that can be used to measure physical activities. Further, weight losing programmes as offered by Weightwatchers rely on recording, sharing and discussing food consumption habits with members of a self-help group. A discussion on the psychological impact of sharing and discussing such information with members of a group is discussed by Weiner (1998).

10.3.3 Reminiscence Therapy

10.3.3.1 Scenario

“Alzheimer! It hit him like a ton of bricks. When Pete started to forget things, he first blamed it on the stress he faced every day. But the more he forgot, the less he felt that it was due to stress. He finally decided to go to a specialist to have his memory checked. Then, the verdict came: We are sorry to inform you that you suffer from an early form of Alzheimer’s disease. Alzheimer! It effects concentration, memory, judgement, rendering its victims helpless over the years. He did not want to forget! He wanted to remember his life! If only there was a way to help him remember his life … .”

10.3.3.2 Comment

Reminiscence enables us to relive events from our past. The American Psychological Association defines reminiscence therapy as “the use of life histories—written or oral, or both—to improve psychological well-being” (VandenBos 2006). It is one of the most popular therapies in dementia care (Woods et al. 2009; Pittiglio 2000). Lifelogging technology has been successfully used to support such reminiscence sessions (Piasek et al. 2011; Bharoucha et al. 2009). The authors of these studies conclude that lifelogging material such as personal pictures can be used as memory trigger to help Alzheimer patients to remember certain events of their life and thus helps to improve their quality of life.

10.3.4 Social Activity Capturing

10.3.4.1 Scenario

“Carina was about to experience the time of her life. Months ago, she received the confirmation that she was selected to participate in the Erasmus programme, i.e., she would be allowed to go abroad and study at another university in Europe for a year. From what she has heard from previous participants of this programme, an Erasmus year was all about exploring the host nation’s country and culture, meeting new people from all over the world and, most of all, party, party, party. Willing to share this experience with her friends at home, she installs a novel lifelogging application on her mobile phone which creates a daily log of her activities, locations and people she meets.”

10.3.4.2 Comment

Technologies, e.g., mobile phone apps, that allow to capture various aspects of the life logger’s life have been presented in Sect. 2.3. Various patents have been granted that lay ground for technology that enable tracking of activity and location (Haner 2002; Olmassakian 1999). A discussion on ethical issues regarding human centric GPS tracking and monitoring is provided by Michael et al. (2006).

The danger of sharing social activity data online is illustrated by Friedland and Sommer (2010), who present an algorithm that can be used to identify potential burglary targets based on the geo-location of videos and pictures that have been shared online by the potential victims. They refer to this as cybercasing, i.e., the method of “using online tools to check out details, make inferences from related data, and speculate about a location in the real world for questionable purposes”.

10.4 Visualization Use Cases

As stated, personal lifelogs are voluminous and complex. Thus, visualization of the lifelog collection should be intuitive, logical and comprehensible. In this chapter, we discuss three use cases of lifelogs visualization. In each subsection talking about each pattern, we give some examples of this pattern. We demonstrate examples of our context-based visualization and hope that this research may invoke further research interest and efforts towards better visualizations for lifelogging content/information.

10.4.1 Visual Diary

A wearable camera passively captures thousands of photos per day. For example, SenseCam takes approximately 5500 images per day. Hence, grouping sequences of related images into events is necessary in order to reduce complexity (Doherty et al. 2008). This event recognition involves the segmentation of all photos into distinct groups, or events, e.g., having breakfast, talking to a work colleague, meeting a friend at a restaurant, etc. To achieve this goal, context-based sensor analysis is required in conjunction with content-based image analysis. Further, representative photos for each event have to be selected, namely a single photo from within an event which represents the event’s content.

We observed that the user interest in visual logs is twofold: to gain an overview, and to find important events of interest. In this section, we present different visualization techniques to best display this information back to the user. Hence, this pattern addresses Use Cases 3.1 (Personal Lifelogging Legacy), 3.3 (Reminiscence therapy) and 3.4 (Social Activity Capturing) where a summary of the lifelogger’s day/week/life is required. The different approaches are introduced in the remainder of this section.

10.4.1.1 Comic-Book Style Visual Diary

A Comic-book style interface requires least space when the screen space is limited. The design is inspired by the Squarified treemap (Bruls et al. 2000) pattern. In order to better utilize the limited screen space, compact view of the visual lifelog is displayed to fit full screen. As shown in Fig. 10.1, it provides a summary of user’s daily visual log on one single page, with emphasis placed on more important events. Each grid represents an event (typically, about 20–25 events are detected for a normal day). The size of the grid provides an immediate visual cue to the events importance level, which offers users a clear entry point for exploration. The position of the grid depicts the time sequence.

Fig. 10.1
figure 1

Interactive visual diary generated for one day, showing event segmentation (reprinted from Jung et al. 2013)

Full size image

Each picture that is displayed in this comic-style visual diary represents multiple similar images that depict the same event. At the same time, users are allowed to drill down (full photo stream and sensor log) inside each event. By clicking on one of the pictures, a user can view these additional pictures, together with a textual description as complementary information. Users tap on other view options to see lifelog data over a different timespan (week/month/year). An example is shown in Fig. 10.2.

Fig. 10.2
figure 2

Each event has multiple images that allows further interaction on demand (reprinted from Jung et al. 2013)

Full size image

10.4.1.2 Timeline (Relive the Day)

While the previous visualization focuses on highlighting important events, the following visualization theme allows users to browse through the entire collection chronologically. It is referred to as “timeline”, i.e., users can see what they have done on a specific day. It can help users to see the relationship between events and comparison across historical data. Firstly, all images are segmented in to different event, then they are displayed on an interactive timeline series. For example, in Fig. 10.3, a list of events is displayed on a timeline for further exploration. For further exploration, users are allowed to access details by clicking or hovering over an event, i.e., users can see what they have done on a specific day. A short text description is generated automatically for each event, which acts as memory for better user experience.

Fig. 10.3
figure 3

Visualization of an interactive timeline

Full size image

10.4.1.3 Master-Detailed

Applying Shneiderman’s mantra of “overview first, zoom and filter, then details-on-demand” (Shneiderman 1996), another visualization pattern allows users to browse a large archive collection in one single place. The visual log is summarized in a master-list (thumbnail) of events and a text abstract representation of each event is shown. This example illustrate a thumbnail gallery to navigate visual life logs. This pattern displays visual lifelogs as a series of thumbnails of grids that users can hover over and zoom in to find more details. Drill-down is supported to reveal a detailed view. An example is shown in Fig. 10.4.

Fig. 10.4
figure 4

Master detailed view of visual log

Full size image

10.4.2 Social Interaction Radar Graph

Our goal is to build a visualization that users could use to facilitate discovery and increased awareness of their social activities. Sensor data from Bluetooth, Wi-Fi, and phone call logs, instant message logs, etc. can be used to gather users’ social interaction information. This social context data is potentially valuable to life memories. Hence, the pattern can be applied in Use Case 3.4 (Social Activity Capturing).

Displaying one type of data along only reflects one aspect of social interaction activity. Aggregating all related sensors together can provide an informative and complete presentation to end-users.

This visual representation, Fig. 10.5, utilizes three embedded sensors’ loggings (i.e., Bluetooth, Wi-Fi, GPS) to automatically identify social context data of individual users over the course of a whole year. Each concentric circle represents one type of sensor data. This radar graph has three dimensions: (1) type of sensor (different color been used), (2) social activity level (length of each area), (3) time (circular radar graph divided into 12 sections, one month per section). Scrolling around the circle enable rapid exploration and comparison between different months. The combination of visual attributes (length and coloring) leverages pre attentive processing to facilitate easy detection of trends or patterns over a year. For example, we can quickly perceive large clumps of gray sections in March, May and October, which tell us that the user experienced higher social activities during these months.

Fig. 10.5
figure 5

Social interaction radar graph (reprinted from Jung et al. 2013)

Full size image

10.4.3 Activity Yearly Calendar

We want to design a visualization that provides users an overview of the whole year on user’s activities, thus addressing Use Case 3.2 (Energy Expenditure Measurement). We focus on the accelerometer and GPS in this visualization because these two types of data are essential to reveal physical movement level directly.

Figure 10.6 shows the Activity view, which allows users gain a detailed understanding of their physical activities. The level of physical activity can be derived from the accelerometer and GPS data. We visualize the data in an annual calendar layout, with color-coding to present the activity intensity. A darker color indicates more activity involved in a given day. By investigating the activity pattern over the course of a full year, it is possible to detect a user’s extreme days (i.e., the most active or the most quite days). This visualization method has a relatively high data-ink ratio, a concept defined by Tufte (1983). For example, in this graph, we can quickly see that the darker green grids are more heavily distributed towards beginning of April, May, July and August, indicating that more activities happened during that time.

Fig. 10.6
figure 6

Activity Calendar View (reprinted from Jung et al. 2013)

Full size image

10.5 Conclusion

Most existing approaches to the presentation of lifelog data use a generic type of user interface (UI) to present all their life-log collection. Given the fact that users may access their lifelogging data under different contexts and goals, we argue that the efficiency and accuracy or user interaction may suffer by relying on a single interface type. In this chapter, we have introduced different HCI design patterns that can be used as guidelines for the development of different lifelogging visualization tools. Therefore, we first introduce different lifelogging technologies that illustrate how this domain has already reached our every-day life. Further, we introduced different use cases that demonstrate different lifelogging scenarios. Based on these scenarios, we then outlined UI templates. We argue that applying these templates can ease the development of lifelogging visualization tools. As future research, we will develop various demonstrator systems based on these templates and aim to design common UI design pattern solutions for the visualization of lifelogging data.