Keywords

1 Introduction

1.1 Hardship in Collecting Data from Experiment

Due to COVID-19, there have been substantial changes in people’s lives including limited in-person contact. It is expected that even after COVID-19 is terminated a number of aspects of our lives will not be the same as before and rather transform into new ways of life. These changes brought a lot of constraints on human-computer interaction or human factors research as well. Particularly, it might be inconvenient or somewhat infeasible to collect human performance data through physical contact with participants in usability test or lab experiment under the pandemic circumstances due to the policy that strongly restricts the physical contact. In order to address this constraint, other experimental methods without physical contact with participants are required.

1.2 Remote Evaluation Methods

Before the constraints of the pandemic, several usability evaluation methods were introduced and used to collect data without physical interaction with participants. For example, Brush et al. demonstrated a remote usability test method. In their study, each participant had to visit the lab to install an experimental prototype in their own computer and data including verbal protocol was collected remotely though phone call [1]. Interestingly, they found no significant differences between in-site and remote test as well as the fact that most participants preferred the remote experiment. Tulis et al. used a web environment for the participant to participate the test in their own place using web browsers [2]. However, there were limitations in data collection due to infeasibility of real-time interaction between experimenter and participant. Andresen et al. conducted a study to compare different testing methods including lab testing, remote synchronous testing, remote asynchronous expert testing, and remote asynchronous user testing [3]. In their remote test methods, videos of participants’ face and screen were recorded along with manual data collection.

Even though there have been many successful instances of remote evaluation methods, there are still problems in the remote data collection methods in the previous studies compared to actual experiment in a lab environment, including: 1) the participants should have the experimental equipment or prototype in own place or should receive from experimenter; 2) it is hard to instruct experimental tasks as well as to monitor the participant performance in real time; 3) types of tasks that the participants are limited; and 4) the types and quality of performance measures (e.g., reaction time, attention movement, etc.) are also limited.

In order to address this constraint from the Pandemic and historical remote test methods, this study proposes an experimental methodology in collecting data through synchronized remote experiments in a simple way. Since a decent video conference software supports remote control feature, it can be used for remote experiment. That is, while an experimental prototype is running in experimenter’s computer, participants can manipulate the prototype using the remote-control functions in their own place rather than in a lab. A case study conducted using the method may validate its utility and demonstrate more detailed ideas.

2 Case Study: Investigating Legibility in Vehicle Interface Using Synchronized Remote Experiment

The original purpose of the study was to investigate effective font faces and line spacing for in-vehicle infotainment display by assessing legibility and satisfaction of drivers in terms of driving safety. Instead of inviting participants to the lab to conduct an experiment with driving tasks by using a driving simulator, an experimental method was used, allowing participants to remotely complete experimental tasks at their residence. The experimental data was collected using a commercial video conference application which provides remote control feature.

That is, the experimental task prototype was executed on the experimenter’s computer and a participant used the remote-control function during the online conference in order to perform the given task, including mouse cursor control and keystrokes. Participants’ performance data was recorded in the prototype application on the experimenter’s computer and the video files of participants’ face including eye movement and reactions recorded by the video conference application. Figure 1 illustrates general structure of the experimental setup and feature of the outputs.

Fig. 1.
figure 1

Conceptual structure of experimental setup for remote task completion

Full size image

As shown in Fig. 1. A Java-based driving task simulation software was presented in experimenter’s computer (i.e., host of the video conference), which includes a primary tracking task and a secondary information acquisition task. During the experiment, a participant was watching the simulator as well as complete the tracking task using keyboard or mouse in his/her own computer. The keyboard or mouse event signals were transmitted to experimenter’s computer through remote control functions in the video conference tool. Then the experimenter’s computer runs based on participants’ manipulations and designated output files were recorded in experimenter’s computer according to the program (e.g., tacking performance such as distance to target along with the time stamps). Since the participant’s computer has video camera, the participant’s face could be recorded in a video file in a cloud or experimenter’s local computer, and this was used to analyze participant’s eye (attention) movement profile. In addition to this, participant’s task completion behavior (e.g., cursor movement) was video recorded for further analysis. All experiment were successfully completed, and viable performance data were collected without inviting the participant to the experiment place.

3 Discussion and Conclusion

Based on the case study, it was possible to confirm the advantages of the remote experiment as followings:

  • Empirical research could be conducted while communicating non-face-to-face with participants without physical contact

  • Participants can reduce the physical and psychological burden of visiting a laboratory by joining in experiments at their residence

  • Experimenters are able to collect and record data easily with functions that the commercial online conference application provides

  • Participants usually have a high understanding on how to use the commercial video conference tool, rather than complicate manual network and system setting.

However, there were some limitations on the methods, including:

  • It was necessary to address every participant’s computer specification (camera, size of monitor and resolution of display) and experiment environments (the view angle depending on participant’s posture, illumination and light). Therefore, the experimenter provided participants with sufficient instruction to setup the participant’s environment before the data collection.

  • Since participant’s personal computers are mainly used, it is difficult to provide a high-fidelity task simulator environment such as full driving simulator. In the case study, the experimental prototype was developed carefully to be manipulated by simple keystroke and mouse events.

  • Network delay occurs occasionally but it was not quite substantial to degrade participants performance and data collection in the case study.

Despite these some shortcomings, the experiment in the case study has been conducted successfully using the method because it seemed that the advantages in the current situation are overwhelming.

This experimental method can be adjusted according to the type or purpose of each of other relevant experiments in future. It is also expected that the method can be actively utilized in various ways as a non-face-to-face data collection method that overcomes physical experimental limitations.