Keywords

1 Introduction

Lifelogging is the process of gathering personal data generated by the activities of an individual. Most of the related work on lifelogging is focused on gathering and analysing visual and physical activity data [1]. In this work we take this further by including HCI tracking and analysis. In our modern society computers have become a core part of our professional and personal environments, which makes capturing HCI activities a rich source of information to mine for generating better insights into our life. It is our conjecture that capturing HCI data is supporting technology for several applications, ranging from simple automatic memory/diary generation, to the discovery of user’s interests, skills, preferences and mood. Personalisation, profiling and security applications are common examples of how such lifelog data can be utilised. We believe that the passive continuous tracking of HCI can reveal important information about the user, which can be used to perform advanced analytics and to build various personalised services. In this work, we present a comprehensive logging tool to capture a wide range of our human-computer interactions.

2 Background

Although one can easily find a commercial spyware/surveillance software providing the ability to monitor and record the computer activities, it is not easy to find dedicated and comprehensive tools for logging computer usage for analysis purposes. In [2], authors discussed using surveillance software for conducting HCI experiments. They described several issues related to time stamps, mouse clicks, web usage and some other missing functionality in such software. Other currently available software is designed to track interaction with one application, for example, WOSIT [3] can be used to observe user actions on one application’s UI in UNIX systems. OWL [4] is another application-specific logging tool that tracks Microsoft Word usage.

Several works have utilised HCI logging for various applications. For example, the use of keystroke logging tools in the domain of cognitive writing research has created new possibilities by providing detailed information about the writing process that was not accessible previously. Particularly, key-loggers are utilised in studies on writing processes, description of writing strategies, children writing development, writing and spelling difficulties analysis [5–7]. Inputlog is an example of such logging tool [8]. Keyboard logging also can be integrated in educational fields for programming and typing skills [9]. In addition, detailed timing of keystrokes can be employed for security purposes [10, 11] and for stress [12] and emotion detection [13]. Screenshots are another example of possible logging technique that can be used to capture and analyse the content consumed or produced by the user. Screenshots were applied for search and task automation [14], and for building user friendly help manuals and demonstrations [15].

Furthermore, intrusion detection and authentication are common applications based on HCI logging. In [16], mouse curves are logged and used as a signature to authenticate users. Several mouse usage features are used to build user profiles [17, 18]. Log files, resource and command usage are used to profile users behaviour [19, 20]. In [21], user identification was explored via logging various GUI interactions such as windows switching frequency, time between new windows and number of opened windows.

Motivated by the beneficial potential and applications of HCI logging, and by the lack of generic and comprehensive HCI tracking and visualisation tools, we present LoggerMan a generic, passive and application-independent logging solution for researchers and lifeloggers. LoggerMan provides a robust tool to assist data gathering, visualisation and analytics for research community.

3 LoggerMan Overview

LoggerMan helps researchers and lifeloggers to collect interaction data produced during normal computer usage. The main goal of LoggerMan is to work passively in the background, intercept usage events and store them for later analysis. It gathers wide range of keyboard, mouse and UI actions. LoggerMan is available online (LoggerMan.org) for interested researchers/lifeloggers. In the next sections, we will describe the system main components and functionality.

3.1 Privacy and Technical Specification

LoggerMan works under Mac OSX 10.7 or later. To avoid any privacy concerns among users, all captured data is stored locally on the computer. This provides the user with a full control of the data. Buffering techniques are used to store the stream of events efficiently to the hard disk. All modules are designed to be Unicode compatible. Thus, the tool can log texts of different languages properly. Loggerman can be configured to run automatically after a system restart. To maintain user privacy, LoggerMan does not log data typed in secured fields (password fields).

Fig. 1.
figure 1

LoggerMan menu

Full size image

3.2 Modules

LoggerMan has been designed to be flexible and modular. Our tool consists of multiple logging modules which can be switched on/off independently (Fig. 1). Each logging module captures a different type of data and every log-entry is timestamped to the current time.

Keyboard-Related Modules. These modules are responsible for capturing keystrokes events and storing them in local files. They provide two levels of detail: word based and key based. Word-based mode (visible as “Keyboard” menu item on Fig. 1) segments and concatenates the events stream into words. Whereas, key-based mode (visible as “Keystrokes” menu item on Fig. 1) tracks each key separately and stores it associated with its time stamp. This particularly important for the domain of keystrokes dynamics [10].

Mouse Module. All mouse actions are traced by this module: move, right/left up/down mouse buttons events, scroll and drag-and-drop actions. We previously utilised the data stored by this module to build an authentication system based on mouse curves [16].

Screenshots Module. This will capture a screenshot of the current active window. The user can select one of the shooting intervals (every 5, 10, 30 s) or a smart-shooting option, which takes a screenshot: after every window transition (from app to app or even different windows in the same app) and at fixed 1 min intervals also. This helps to ensure a reasonable trade-off between capture frequency and storage usage. The screenshot can then provide valuable information about the content that the user is consuming or non-texual content that the user is creating.

Apps Module. This module is designed to track apps transitions. It only logs the currently used app regardless of how it is being used, and as such it offers a simple overview of app usage, without any potential privacy issues of actually capturing the screens.

Clipboard Module. Clipboard module is responsible for tracking copy-paste operations. Any text the user copy to the clipboard is captured and logged.

3.3 Reporting and Insight

The current version of LoggerMan comes with a local reporting tool to visualise and give insights to the data owner about his/her computer usage. Specifically, the reporting tool presents three tag clouds: used apps, typed words and windows titles. These clouds are constructed by computing the normalised frequencies of items occurrence during the selected date range by the user (Figs. 2 and 4). Information about mouse clicks and typed keystrokes during a time frame is also presented as a graph (Fig. 3). In addition, LoggerMan organises the screenshots and the app usage data in a zoom-able timeline to provide a dyanmic and up-to-date view of computer usage (Fig. 5).

Fig. 2.
figure 2

Apps cloud

Full size image
Fig. 3.
figure 3

Mouse/keyboard usage

Full size image
Fig. 4.
figure 4

Windows titles cloud

Full size image
Fig. 5.
figure 5

Apps & screenshots timeline

Full size image

4 Demonstration and Evaluation

LoggerMan is designed to be easy to install and run by any user. The user is all set by just double clicking the app icon for the first time. After that, the app will keep working automatically and passively in the background. Three subjects have been using LoggerMan to capture their computer usage for more than 2 months. Although the data size is dependent of the running modules and the computer usage intensity, LoggerMan produces around 1 GB of data per week for each user with average computer usage. Most of this space (98 %) is taken by screenshots. During MMM 2016, our demo will be conducted in a demo mode where conference participants will be able to log their interactions for a short period of time, then they can visualise their data by the reporting module of LoggerMan. Participants can get a copy of their interaction log files before the data being deleted at the end of the session. A personalized report will be generated for each user to show their uniqueness compared to other users.

5 Conclusions

There is a lack of a useful logging tools in the domain of human-computer interaction. In this paper, we presented LoggerMan, a logging and visualisation tool to capture computer activities and to give some insight based on this data. LoggerMan can easily support a wide range HCI related research such as user identification, personalisation, recommendation and cognition.