Abstract
As we become increasingly dependent on our computers and spending a major part of our day interacting with these machines, it is becoming important for lifeloggers and Human-Computer Interaction (HCI) researchers to capture this aspect of our life. In this paper, we present LoggerMan, a comprehensive logging tool to capture many aspects of our computer usage. It also comes with reporting capabilities to give insights to the data owner about his/her computer usage. In this work, we aim to address the current lack of logging software in this domain, which would help us, and other researchers, to build datasets for HCI experiments and also to better understand computer usage patterns. Our tool is published online (http://loggerman.org/) to be used freely by the community.
Access provided by Autonomous University of Puebla. Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Lifelogging is the process of gathering personal data generated by the activities of an individual. Most of the related work on lifelogging is focused on gathering and analysing visual and physical activity data [1]. In this work we take this further by including HCI tracking and analysis. In our modern society computers have become a core part of our professional and personal environments, which makes capturing HCI activities a rich source of information to mine for generating better insights into our life. It is our conjecture that capturing HCI data is supporting technology for several applications, ranging from simple automatic memory/diary generation, to the discovery of user’s interests, skills, preferences and mood. Personalisation, profiling and security applications are common examples of how such lifelog data can be utilised. We believe that the passive continuous tracking of HCI can reveal important information about the user, which can be used to perform advanced analytics and to build various personalised services. In this work, we present a comprehensive logging tool to capture a wide range of our human-computer interactions.
2 Background
Although one can easily find a commercial spyware/surveillance software providing the ability to monitor and record the computer activities, it is not easy to find dedicated and comprehensive tools for logging computer usage for analysis purposes. In [2], authors discussed using surveillance software for conducting HCI experiments. They described several issues related to time stamps, mouse clicks, web usage and some other missing functionality in such software. Other currently available software is designed to track interaction with one application, for example, WOSIT [3] can be used to observe user actions on one application’s UI in UNIX systems. OWL [4] is another application-specific logging tool that tracks Microsoft Word usage.
Several works have utilised HCI logging for various applications. For example, the use of keystroke logging tools in the domain of cognitive writing research has created new possibilities by providing detailed information about the writing process that was not accessible previously. Particularly, key-loggers are utilised in studies on writing processes, description of writing strategies, children writing development, writing and spelling difficulties analysis [5–7]. Inputlog is an example of such logging tool [8]. Keyboard logging also can be integrated in educational fields for programming and typing skills [9]. In addition, detailed timing of keystrokes can be employed for security purposes [10, 11] and for stress [12] and emotion detection [13]. Screenshots are another example of possible logging technique that can be used to capture and analyse the content consumed or produced by the user. Screenshots were applied for search and task automation [14], and for building user friendly help manuals and demonstrations [15].
Furthermore, intrusion detection and authentication are common applications based on HCI logging. In [16], mouse curves are logged and used as a signature to authenticate users. Several mouse usage features are used to build user profiles [17, 18]. Log files, resource and command usage are used to profile users behaviour [19, 20]. In [21], user identification was explored via logging various GUI interactions such as windows switching frequency, time between new windows and number of opened windows.
Motivated by the beneficial potential and applications of HCI logging, and by the lack of generic and comprehensive HCI tracking and visualisation tools, we present LoggerMan a generic, passive and application-independent logging solution for researchers and lifeloggers. LoggerMan provides a robust tool to assist data gathering, visualisation and analytics for research community.
3 LoggerMan Overview
LoggerMan helps researchers and lifeloggers to collect interaction data produced during normal computer usage. The main goal of LoggerMan is to work passively in the background, intercept usage events and store them for later analysis. It gathers wide range of keyboard, mouse and UI actions. LoggerMan is available online (LoggerMan.org) for interested researchers/lifeloggers. In the next sections, we will describe the system main components and functionality.
3.1 Privacy and Technical Specification
LoggerMan works under Mac OSX 10.7 or later. To avoid any privacy concerns among users, all captured data is stored locally on the computer. This provides the user with a full control of the data. Buffering techniques are used to store the stream of events efficiently to the hard disk. All modules are designed to be Unicode compatible. Thus, the tool can log texts of different languages properly. Loggerman can be configured to run automatically after a system restart. To maintain user privacy, LoggerMan does not log data typed in secured fields (password fields).
3.2 Modules
LoggerMan has been designed to be flexible and modular. Our tool consists of multiple logging modules which can be switched on/off independently (Fig. 1). Each logging module captures a different type of data and every log-entry is timestamped to the current time.
Keyboard-Related Modules. These modules are responsible for capturing keystrokes events and storing them in local files. They provide two levels of detail: word based and key based. Word-based mode (visible as “Keyboard” menu item on Fig. 1) segments and concatenates the events stream into words. Whereas, key-based mode (visible as “Keystrokes” menu item on Fig. 1) tracks each key separately and stores it associated with its time stamp. This particularly important for the domain of keystrokes dynamics [10].
Mouse Module. All mouse actions are traced by this module: move, right/left up/down mouse buttons events, scroll and drag-and-drop actions. We previously utilised the data stored by this module to build an authentication system based on mouse curves [16].
Screenshots Module. This will capture a screenshot of the current active window. The user can select one of the shooting intervals (every 5, 10, 30 s) or a smart-shooting option, which takes a screenshot: after every window transition (from app to app or even different windows in the same app) and at fixed 1 min intervals also. This helps to ensure a reasonable trade-off between capture frequency and storage usage. The screenshot can then provide valuable information about the content that the user is consuming or non-texual content that the user is creating.
Apps Module. This module is designed to track apps transitions. It only logs the currently used app regardless of how it is being used, and as such it offers a simple overview of app usage, without any potential privacy issues of actually capturing the screens.
Clipboard Module. Clipboard module is responsible for tracking copy-paste operations. Any text the user copy to the clipboard is captured and logged.
3.3 Reporting and Insight
The current version of LoggerMan comes with a local reporting tool to visualise and give insights to the data owner about his/her computer usage. Specifically, the reporting tool presents three tag clouds: used apps, typed words and windows titles. These clouds are constructed by computing the normalised frequencies of items occurrence during the selected date range by the user (Figs. 2 and 4). Information about mouse clicks and typed keystrokes during a time frame is also presented as a graph (Fig. 3). In addition, LoggerMan organises the screenshots and the app usage data in a zoom-able timeline to provide a dyanmic and up-to-date view of computer usage (Fig. 5).
4 Demonstration and Evaluation
LoggerMan is designed to be easy to install and run by any user. The user is all set by just double clicking the app icon for the first time. After that, the app will keep working automatically and passively in the background. Three subjects have been using LoggerMan to capture their computer usage for more than 2 months. Although the data size is dependent of the running modules and the computer usage intensity, LoggerMan produces around 1 GB of data per week for each user with average computer usage. Most of this space (98 %) is taken by screenshots. During MMM 2016, our demo will be conducted in a demo mode where conference participants will be able to log their interactions for a short period of time, then they can visualise their data by the reporting module of LoggerMan. Participants can get a copy of their interaction log files before the data being deleted at the end of the session. A personalized report will be generated for each user to show their uniqueness compared to other users.
5 Conclusions
There is a lack of a useful logging tools in the domain of human-computer interaction. In this paper, we presented LoggerMan, a logging and visualisation tool to capture computer activities and to give some insight based on this data. LoggerMan can easily support a wide range HCI related research such as user identification, personalisation, recommendation and cognition.
References
Gurrin, C., Smeaton, A.F., Doherty, A.R.: LifeLogging: personal big data. Found. Trends Inf. Retr. 8(1), 1–125 (2014)
Liffick, B.W., Yohe, L.K.: Using surveillance software as an HCI tool. In: Information Systems Education Conference (2001)
The MITRE corporation. Widget observation simulation inspection tool. http://www.openchannelfoundation.org/projects/WOSIT. Accessed 07 Aug 2015
Frank, L.: OWL: A recommender system for it skills
Sullivan, K.P.H., Lindgren, E.: Computer Key-Stroke Logging and Writing: Methods and Applications. Studies in Writing, vol. 18. Elsevier, Oxford (2006)
Leijten, M., Van Waes, L.: Keystroke logging in writing research: using inputlog to analyze and visualize writing processes. Writ. Commun. 30(3), 358–392 (2013)
Stromqvist, S., Holmqvist, K., Johansson, V., Karlsson, H., Wengelin, A.: What keystroke-logging can reveal about writing. computer key-stroke logging and writing: methods and applications. Stud. Writ. 18, 45–72 (2006)
Leijten, M., Van Waes, L.: Inputlog: a logging tool for the research of writing processes. Technical report
Rodrigues, M., Gonçalves, S., Carneiro, D., Novais, P., Fdez-Riverola, F.: Keystrokes and clicks: measuring stress on e-learning students. In: Casillas, J., Martínez-López, F.J., Vicari, R., De la Prieta, F. (eds.) Management Intelligent Systems. AISC, vol. 220, pp. 119–126. Springer, Heidelberg (2013)
Gunetti, D., Picardi, C.: Keystroke analysis of free text. ACM Trans. Inf. Syst. Secur. 8(3), 312–347 (2005)
Shimshon, T., Moskovitch, R., Rokach, L., Elovici, Y.: Clustering di-graphs for continuously verifying users according to their typing patterns. In: 2010 IEEE 26th Convention of Electrical and Electronics Engineers in Israel (IEEEI), pp. 000445–000449 (2010)
Vizer, L.M.: Detecting cognitive and physical stress through typing behavior. In: CHI 2009 Extended Abstracts on Human Factors in Computing Systems, pp. 3113–3116. ACM (2009)
Kolakowska, A.: A review of emotion recognition methods based on keystroke dynamics and mouse movements. In: 2013 the 6th International Conference on Human System Interaction (HSI), pp. 548–555 (2013)
Yeh, T., Chang, T.-H., Miller, R.C.: Sikuli: Using GUI screenshots for search and automation. In: Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, UIST 2009, pp. 183–192. ACM (2009)
Yeh, T., Chang, T.-H., Xie, B., Walsh, G., Watkins, I., Wongsuphasawat, K., Huang, M., Davis, L.S., Bederson, B.B.: Creating contextual help for GUIs using screenshots. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 145–154 (2011)
Hinbarji, Z., Albatal, R., Gurrin, C.: Dynamic user authentication based on mouse movements curves. In: He, X., Luo, S., Tao, D., Xu, C., Yang, J., Hasan, M.A. (eds.) MMM 2015, Part II. LNCS, vol. 8936, pp. 111–122. Springer, Heidelberg (2015)
Ahmed, A.A.E., Traore, I.: A new biometric technology based on mouse dynamics. IEEE Trans. Dependable Sec. Comput. 4(3), 165–179 (2007)
Pusara, M., Brodley, C.E.: User re-authentication via mouse movements. In: Proceedings of the 2004 ACM Workshop on Visualization and Data Mining for Computer Security, pp. 1–8. ACM (2004)
Anderson, J.P.: Computer Security Threat Monitoring and Surveillance. James P. Anderson Co. (2002)
Yeung, D.Y., Ding, Y.: Host-based intrusion detection using dynamic and static behavioral models. Pattern Recognit. 36, 229–243 (2003)
Goldring, T.: User profiling for intrusion detection in windows nt. Comput. Sci. Stat. 35 (2003)
Acknowledgments
This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/12/RC/2289.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Hinbarji, Z., Albatal, R., O’Connor, N., Gurrin, C. (2016). LoggerMan, a Comprehensive Logging and Visualization Tool to Capture Computer Usage. In: Tian, Q., Sebe, N., Qi, GJ., Huet, B., Hong, R., Liu, X. (eds) MultiMedia Modeling. MMM 2016. Lecture Notes in Computer Science(), vol 9517. Springer, Cham. https://doi.org/10.1007/978-3-319-27674-8_31
Download citation
DOI: https://doi.org/10.1007/978-3-319-27674-8_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27673-1
Online ISBN: 978-3-319-27674-8
eBook Packages: Computer ScienceComputer Science (R0)