Keywords

1 Introduction

Over the past decades, many healthcare professionals have identified an urgent need to reduce the cost and increase the accessibility of healthcare practices. Internet-based healthcare applications directly address the need and change many aspects of healthcare and wellness [1, 2]. The advent of online services and the ubiquity of mobile devices have enabled a paradigm shift in how people use healthcare and wellness services from face-to-face interactions to Internet-based online interaction. The online healthcare practices allow participants to access clinical interventions at a lower cost and at their convenience. In addition, most of the applications support anonymous participation in their interventions. As a result, the new form of healthcare service has significantly enhanced the Nation’s health by improving knowledge on patients’ behaviour and health outcomes [3, 4].

Despite the promising future of online healthcare applications, its growth rate is slow to moderate. One of the biggest obstacles is the lack of systematic evaluation platform for online healthcare applications. Among numerous applications, only a few applications have been evaluated [5] and it reveals an urgent need for the rigorous evaluation of these applications for safety and health. However, there is no standard platform or tool exists. Without stringent evaluations, it is impossible for healthcare professionals to check if participants use their applications in a correct way and their applications are effective as designed. If patients keep using online healthcare application with unproven efficiency and reliability, it quickly ruins the applications’ reputation and, in turn, reduces the size of the online healthcare market.

To address issues pointed above, we need a software system to engineer the evaluation of online health and wellness applications as a general-purpose solution for the market. The evaluation system must provide the four services below.

  1. 1.

    Systematic authoring of online clinical studies to measure the correctness and effectiveness of online healthcare applications.

  2. 2.

    Automatic delivery of assessments and/or interventions to participants according to a study’s workflow.

  3. 3.

    Privacy-preserving online monitoring while participants conduct online assessments and/or interventions on healthcare applications. It is critical that PESOHA considers the privacy and security of FDA-monitored clinical research trials on online healthcare applications.

  4. 4.

    Providing study-related monitoring data for the evaluation of online healthcare applications and clinical studies.

To meet the requirements listed above, in this paper, we propose the privacy-preserving evaluation system for online healthcare applications, in short PESOHA. The proposed system helps us overcome cost, technological and intellectual barriers to rigorous evaluation, thereby accelerating translation from existing applications with unknown efficiency to online applications proven effective in improving health. PESOHA enables non-IT healthcare professionals to create their clinical studies on online applications without knowledge and skills on information technologies, in order to measure the performance of a wide range of online healthcare applications including weight control applications, smoking cessation applications, diabetes control applications, stress management applications, and mental treatment applications. By analysing monitoring data collected by PESOHA, they can optimize their healthcare applications and improve the applications’ quality of service. The innovative evaluation method of PESOHA will expand the role of online healthcare applications to improve our Nation’s health and wellness.

The rest of this paper is organized as follows. In Sect. 2, we define several concepts and present the structural architecture of PESOHA in Sect. 3. To demonstrate PESOHA’s services, in Sect. 4, we present two operational flows for medical researchers and patients with screenshots. In Sect. 5, we introduced related work, and in Sect. 6, we summarize our contributions and suggest future work.

2 Design of Concepts

In this section, we define some building blocks of PESOHA and present its structural design. We explain each component in the system and relationships between components in detail. For a better understanding of PESOHA’s services, we present two workflows, one for healthcare professionals and the other one for participants in online clinical studies. PESOHA is built upon several concepts, such as user, study, intervention, assessment, and workflow. It is worthy of defining those concepts before we describe the structural design and operations of PESOHA. The definitions of the concepts are described below.

2.1 User

PESOHA categorizes users based on their roles as shown in Fig. 1. A system administrator is in charge of maintaining the PESOHA system. A researcher is a healthcare professional who wants to create clinical studies and evaluate online applications by using PESOHA. A researcher supervises study staff and activity staff. A study staff can create and maintain clinical studies while an activity staff manages study activities (e.g. assessments or interventions) that belong to a study. Study participants, target objects of a particular study, are divided to two different types, advisor and recipient. An advisor is a person who helps patients conduct clinical studies and provides some feedback on patients’ performance. In most cases, family members of patients take advisor roles. A recipient is a patient who participates in a study to improve his/her health. Usually, the recipients are the customers of a target healthcare application.

Fig. 1
figure 1

Role hierarchy in PESOHA

Full size image

A user has eight properties: UserID, Name, Address, Email, Phone, Registration date, Roles, and Permissions. According to the role that a user is taking, few more properties could be added. For examples, a Researcher has an additional property, Study, to represent ownership of a clinical study. A recipient has two more properties, Status and Enrolment history. The Status property represents whether a patient is participating in a study while the Enrolment history is a log of previous participations in past studies.

2.2 Study

A researcher can create a study on online applications to conduct clinical research and evaluate the performance of the applications. To create a study, a researcher must define eight mandatory properties; StudyID, Name, Period, Status, Study staff, Reuse history, Recipient groups, and Workflows. In addition, he/she can define three optional properties including Eligibility rules, Assignment rules, and Consent questions. A study must have a unique ID and name. The Period is specified with a begin date and an end date of a study. A study will be available only during the period. To represent the Status of a study, we define four statuses: Pending, Activated, Inactive, and Deleted as shown in the lifecycle of study in Fig. 2.

Fig. 2
figure 2

Lifecycle of online clinical study, intervention, and assessment

Full size image

When a study is created, its status is ‘Pending’. At the begin date of the study, its status is transmitted to ‘Active’. Note that a study cannot be modified once activated. If the study finishes its execution (Normal Deactivation), or an administrator or a study creator forces to deactivate the study (Abnormal Deactivation), the status changes to ‘Inactive’. An inactive study is reactivated if the begin date is changed to the present (Active) or a date in the future (Pending). Once an active study completes all the activities, the status changes to ‘Deleted’. If a creator or administrator no longer needs an inactive study, he/she can delete the study. A study must have one or more staff including study creators who make the study and study coordinators who manage an overall workflow of the study. If a study creator reuses an existing study created in the past, the information is saved in the Reuse history.

To conduct a study, a study must have at least one recipient group, called Study Arm, and each arm must have own Study Workflow (see Sect. 2.4 for more details). To filter unqualified participants out and assign qualified one to a particular arm, a study may have its Eligibility rules and Assignment Rules, respectively. If a clinical study deals with sensitive and private data, a creator must specify questions in the Consent questions to get prior consent.

2.3 Intervention and Assessment

A study consists of two types of activities: Intervention and Assessment. The goal of an intervention is to change the health status of recipients. An intervention has nine properties: InterventionID, Name, URL, Status, Period, Intervention staff, Reuse history, Intervention Type, and Monitoring data types. Most of the properties are similar to the properties of study, except for a few new properties.

PESOHA handles online interventions and assessments only; thereby, it needs a web address, which is represented as a URL. According to its purpose, an intervention’s type is defined among four types: Text tutorial, Essay, Video tutorial, and Custom Intervention. A text tutorial is a reading material (e.g. reading useful tips for weight control), while a video tutorial is a visual material, such as animations or movie clips (e.g. antismoking campaigns movie clip and online tutorial for teenager driving). Writing an essay could be an intervention, e.g. writing diary or notes about health status. A researcher can create a custom intervention as a sequence of existing interventions. PESOHA collects different types of data depending on the type of an intervention as shown in Table 1, and the Monitoring data types will be specified in the profile of an intervention.

Table 1 Monitoring data types for each intervention type
Full size table

An assessment, another type of study activity, aims at measuring status of a recipient. An assessment has nine properties: AssessmentID, Name, URL, Status, Period, Assessment staff, Reuse history, Assessment type, and Monitoring data types. Most properties are the same as those of interventions, but assessments have different types: Survey, Test, External data, External device, and Custom assessment.

A researcher can assign a survey to recipients to know his/her thoughts and opinion or test (e.g. quiz) to measure improvement in recipients’ health status. PESOHA can import recipients’ health-related data from an external database (external data) or store streaming data from wearable health devices (e.g. KardiaBand for heartbeat tracking and TempTraq for body temperature tracking) (external device). Besides four basic assessment types, a researcher can create a Custom assessment. The type of monitoring data is determined based on an intervention’s type as shown in Table 2.

Table 2 Monitoring data types for each assessment type
Full size table

2.4 Workflow

As mentioned above, each study arm has its own workflow. A workflow consists of several workflow steps and must have a start step and an end step. A workflow step is defined with an associated Activity, a Completion condition, and one or more Post actions. An activity that a study participant needs to perform is either an intervention or an assessment. To check if a participant successfully finishes an assigned activity, PESOHA uses Completion condition. In the Post action, we specify one or more next step(s). To move to the next step, the corresponding Transition condition of the Post action must be satisfied. In most cases, the result of the previous step(s) determines the transition from a step to another.

To specify the order of steps, PESOHA supports four types of routing: sequential, parallel, choice, and iteration. In a sequential routing, several activities are executed in sequence (e.g. ‘First Step A then Step B’). To perform two or more activities at the same time or in any order, we use a parallel routing that commences with an AND-Split and concludes with an AND-Join. An AND-Split allows multiple activities to start their operation simultaneously, and an AND-Join converges parallel executing activities into a single common activity. If a researcher needs to synchronize multiple activities at a certain point, he/she must use AND-Join. Unlike the parallel routing, a choice routing uses OR-Split and OR-Join. In an OR-Split, a single activity among multiple alternatives is selected depending on a corresponding Transition condition. An OR-Join converges multiple alternative activities to a single activity without synchronization of results. An iterative routing involves the repetitive execution of one or a group of step(s) until a transition condition is met. For a better understanding of study workflow, we present two example workflows in Fig. 3.

Fig. 3
figure 3

Example workflows for two groups of study recipients

Full size image

3 Architecture

PESOHA has three layers: application layer, service layer, and database layer. The application layer is a front end of PESOHA. It receives user requests and inputs and passes to the service layer. In addition, it delivers services provided by the service layer to users. All the data required to perform tasks in the service layer are stored in a relational database in the database layer. The overall architecture of PESOHA is shown in Fig. 4.

Fig. 4
figure 4

Structural architecture of PESOHA

Full size image

To fulfill user requests, the service layer has six manager modules: User Manager, Study Manager, Workflow Manager, Intervention Manager, Assessment Manager, and Monitoring Manager. The detailed descriptions for each component are below.

  • User Manager—This module authenticates valid users and handles user information through user creation, modification, and deletion. An administrator can create a user, or a user can register himself/herself in PESOHA (User Registration). Once a user account is created, only administrators can modify and delete the user account. To access PESOHA, a user can log in with his/her account credential issued by PESOHA or use external user authentication services [e.g. OpenID services and single sign-on (SSO) services]. Currently, PESOHA supports Google Sign-In and Facebook SSO.

  • Intervention Manager and Assessment Manager—These two modules are responsible for creating, modifying, and deleting online interventions and assessments. Researchers and administrators can create a study activity, either an intervention or an assessment, by entering details for an activity’s properties. If PESOHA has a similar activity already, they can reuse a past activity. Once an activity is created, these modules must maintain the status of the activity according to the lifecycle shown in Fig. 2. Note that a creator can modify and delete his/her own interventions and assessments while administrators can manage all the activities in PESOHA.

  • Study Manager—This module creates, modifies, and deletes studies by interacting with other managers. To create a study, the Study Manager first needs to receive information about all the participants in a new study from the User Manager. Once a study becomes active, it fetches one or more corresponding workflow(s) from the Workflow Manager. To carry on a study, it delivers interventions and assessments to participants according to a workflow assigned to a participant’s study arm.

  • Workflow Manager—This module maintains all the study workflows. A study creator and administrator can create, modify, and delete workflows. As explained earlier, each study arm must have its own workflow and a workflow step has an associated activity. To get information about interventions and assessments that are linked to a workflow, the Workflow Manager communicates with the Intervention Manager and the Assessment Manager.

  • Monitoring Manager—If the Study Manager delivers a link to an activity, participants access an online intervention or assessment at their convenience. Once a participant starts to use, some usage and/or user data are collected by online monitoring code embedded in the intervention/assessment and sent to the Monitoring Manager in PESOHA. The types of data to be monitored are determined based on the type of activity (see Tables 1 and 2). As mentioned earlier, PESOHA must protect the privacy of patients who are using online healthcare and wellness applications. Towards this, PESOHA uses the privacy-preserving online monitoring (PPoM) service, an online monitoring service that gathers authorized user/usage data that users allow to monitor only [6, 7]. The PPoM service allows participants to specify their privacy preferences on data monitoring. It means that patients can determine which data can be monitored. Then, the PPoM Service selectively collects data based on patients’ preference, not preferences of healthcare service providers.

4 Demonstration

PESOHA provides different services to researchers and participants as shown in Figs. 5 and 6, respectively. For researchers, it provides intuitive user interfaces to create and run a clinical study. In addition, it provides monitoring data related to the study for further analysis of performance and usability of online healthcare applications. Once a researcher creates a study, PESOHA sends an invitation to all the potential participants. If a potential participant accepts the invitation, PESOHA starts an operation for them. Note that online health applications must meet the following requirements to use PESOHA.

Fig. 5
figure 5

Activity model for researchers

Full size image
Fig. 6
figure 6

Activity model for participants

Full size image

  • Must support HTML 4 or higher and Flash to call JavaScript functions.

  • Must have the membership-based authorization.

  • Must define all the HTML objects to be monitored with unique object IDs.

  • Must embed the PPoM monitoring code [6] into each webpage to be monitored.

To participate in studies operated by PESOHA, a participant’s browser must have the following capability.

  • Must enable JavaScript.

  • For mobile browsers, must support JQuery.

  • Must allow HTTP POST method and make sure that Firewall does not block the HTTP POST messages.

4.1 Operation of Researchers

A researcher who is registered in PESOHA can create a clinical study to evaluate online health applications by analysing patients’ usage and user data. Towards this, a researcher logs in PESOHA and clicks the ‘add’ button on the ‘STUDIES’ tab, as shown in Fig. 7. To create a new study, he/she needs to enter all the required data, including Study Name, Status, Begin Date, End Date, IRB approval status, Study Staff (e.g. creators and coordinators), Eligibility Rule, Assignment Rules, Study Arms, and Participants. After creating a Participants Pool, a researcher can define a Study Arm with a subset of study participants and a corresponding workflow. To create a workflow, a researcher first creates all the workflow steps and then specifies routing types for each step. For a sequential routing, we need to specify one next activity as Post Action. For a parallel routing, you must identify an AND-Join and an AND-Split step. If a workflow step is a converged activity that synchronizes all the results from multiple previous steps, you must define the step as an AND-Join step by marking on the ‘AND-Join’ checkbox shown in Fig. 8. To represent an AND-Split step, you must specify two or more next activities by adding several Post Actions. To represent a choice routing, you need to add two or more Post Actions and each Post Action must have a different transition condition. To identify an OR-Join step, one or more activities must have the same single Post Action without marking the ‘AND-Join’ checkbox. For an Iteration routing, the Post Action of an activity must be the activity itself.

Fig. 7
figure 7

Study creation in PESOHA

Full size image
Fig. 8
figure 8

Creation of a workflow activity in PESOHA

Full size image

When assigning participants to a study arm, a researcher can choose one method among three: (1) Manual assignment, (2) Rule-based assignment, or (3) Randomized assignment. The Manual assignment is performed by study staff, while the Rule-based and the Randomized assignments are performed by PESOHA. At the Begin Date, PESOHA runs a study and starts to deliver study activities to participants according to an assigned workflow. If a study is complete, study staff can download all the monitoring data collected for the study. To do so, staff first retrieves the study by clicking the study name on the study navigation panel on the right side (View Study in the Fig. 5) and then clicks the ‘Export Data’ next to the ‘Study Name’.

4.2 Operation of Patients

A patient who wants to participate in a study must register in PESOHA first. Once a patient completes registration, PESOHA stores the patient’s profile in the database, checks his/her login credential, and verifies the patient’s eligibility for a study based on the Eligibility rules that a study creator specified. If a study has an activity that handles sensitive data, PESOHA requires the patient to sign in a consent form. If a patient gives consent, PESOHA assigns the patient into the study as a recipient and starts to send emails containing links to online interventions or assessments. Once a patient conducts all the activities in a study, PESOHA informs him/her of the end of the study and changes the Status of the patient in the database.

5 Related Work

With the advent of seamless network connectivity and Internet-based rich interaction methods (e.g. messaging, posting, chatting, file sharing, and gaming), many online healthcare and wellbeing applications have been developed. Currently, online healthcare applications have been used for a wide range of purposes including medical research [8], remote education [9], online counseling [10], participant recruitment [11], health promotion [12], and Internet-based treatment [13].

To verify the effectiveness of such applications and improve the quality of services, it is critical to evaluate the online healthcare applications. Sherrington et al. [14] reviewed Internet-delivered medical interventions for obese patients. They found twelve medical studies and evaluated nine databases by manually identifying relevant keywords, such as Internet, web, online, eHealth, nutrition, diet, weight, weight loss, overweight, obesity, and clinical trial. After gathering the required data, they analysed weight loss at third, sixth, and twelfth month by calculating the retention rates for each intervention using BMI values. This work evaluated online health applications, but for a specific type of applications (obesity control applications) only. In addition, they manually defined the search keywords and analysis criteria such as retention rates.

Kushniruk et al. [15] evaluated the usability of online health applications by analysing usage data (e.g. page visits and click events). However, this work focused on measuring usability and do not consider the effectiveness of the applications. In 2017, Rogers et al. [16] evaluated the availability of Internet-delivered health interventions using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [17]. They identified currently available health interventions on mental health, weight control, disease prevention and management, and childhood health management, but do not consider the effectiveness of each intervention (e.g. how much does an intervention improve the health of target patients?).

As shown above, most of the evaluations have been limited to specific applications or collected data by collaboration with IT experts. If health professionals use PESOHA, they can easily create a clinical study to evaluate the correctness and the effectiveness of an online assessment or intervention without any help from IT professionals. In addition, they can receive all the data, not only usage data but also user data including health-related data without concern of privacy loss.

6 Conclusion

There is an urgent need for an evaluation system for prevalent online healthcare and wellness applications to verify the performance of the applications and, in turn, improve the quality of online healthcare services. To address the need, we propose a privacy-preserving evaluation system for online healthcare applications, in short PESOHA, which enables healthcare professionals to do the following tasks:

  • Intuitive creation of clinical studies consisting of online interventions and assessments to evaluate the correctness and effectiveness of online healthcare applications.

  • Automatic delivery of created studies to patients based on the patients’ status.

  • Privacy-preserving online monitoring on online interventions and assessments to collect patients’ user data, including health-related data, as well as usage data.

  • Completion check for all recipients and downloading monitoring data for clinical studies.

To verify the usefulness of PESOHA, we plan to conduct a field test with many existing online healthcare and wellness applications. Once we obtain acceptable results from the test, we plan to industrialize PESOHA as a general-purpose evaluation platform for e-health applications. To extend its service to mobile applications, we first need to develop a mobile monitoring service that concerns patients’ privacy preference and then upgrade PESOHA to deal with not only online applications but also mobile applications.