Keywords

1 Introduction

Asthma is a variable long-term condition affecting 339 million people worldwide [1]. There are often diurnal, seasonal, and life-time variations in the symptoms experienced by a patient. Common symptoms include shortness of breath, wheezing, and cough. Asthma attacks, if not treated promptly, can lead to hospitalization or even death [2, 3]. Currently, there is no cure for asthma, so the focus is on patients’ self-management of their condition [4]. This involves monitoring asthma status to inform the best course of action.

Regular monitoring of asthma symptoms may identify worsening asthma status early and action can be taken to avoid further deterioration. However, patients may consider long-term monitoring as tedious, especially during extended periods when they do not experience symptoms, which may lead to a loss of engagement [5].

Mobile health (mHealth) can support asthma home monitoring and asthma self-management through the use of devices such as smartwatches which require much lower levels of active engagement from patients [6]. This ‘passive’ approach has the potential to support many more patients in monitoring and making decisions about their health.

We are currently working towards building a system to monitor asthma without requiring high levels of active engagement, the Asthma Attack Management Online System (AAMOS). It is currently unclear whether passive monitoring would be beneficial or indeed provide higher levels of engagement and compliance in long-term monitoring. As a starting point, we conducted the AAMOS-00 study “Predicting asthma attacks using connected mobile devices and machine learning”, a pilot study focused on collecting novel monitoring data (see [7] for additional details on the study design). The novel data collected during the AAMOS-00 study provides an opportunity to explore compliance with passive monitoring, the focus of this paper.

The primary aim of this study is to test whether passive monitoring would lead to higher compliance over an extended time when compared to active monitoring. The secondary aim is to investigate the usability of the system.

2 Methods

2.1 Study Design

We recruited 32 asthma patients across the United Kingdom who had experienced at least one severe asthma attack (as defined by the American Thoracic Society and European Respiratory Society [2]) during the past year. We undertook the observational study from April 2021 to June 2022 in two phases. During phase one, monitoring was by daily questionnaire over one month to select patients with at least 50% compliance. These patients (n = 22) were then invited to participate in phase two that consisted of device and daily questionnaire monitoring over six months. In addition to using their own smartphone, participants were provided with three smart monitoring devices: a smartwatch (MiBand3 by Xiaomi [8]), a smart peak flow meter (Smart Peak Flow Meter by Smart Asthma [9]), and a smart inhaler (FindAir ONE by FindAir [10]) [7]. Figure 1 provides an overview of the whole research data collection system.

In phase two, participants answered questionnaires at home using the Mobistudy app (a mobile-based research platform for data collection [11]). Participants uploaded data from the smartwatch weekly via a Bluetooth connection, and conducted two sets of peak flow measurements with the smart peak flow meter, once in the morning and once at night; the maximum of three measurements was reported per set. Moreover, participants used the FindAir app to upload data from the smart inhaler. At the end of the AAMOS-00 study, participants were asked to digitally complete an exit questionnaire at home about the acceptability and usability of the study’s data collection system.

Fig. 1.
figure 1

(adapted from our previous publication [7]).

AAMOS-00 system overview

Full size image

Our analysis used data collected from the 22 participants during phase two of the AAMOS-00 study and the end-of-study questionnaire. In particular, the focus was on investigating passive and active monitoring using the daily asthma diary, smart peak flow meter, and smartwatch usage data. The daily asthma diary and smart peak flow meter monitoring tasks reflected current practice of asthma monitoring, while smartwatch use represented a promising technology for passive monitoring.

2.2 Measure of Compliance

Compliance with each monitoring task was defined as their completion on a daily basis. Specifically, daily compliance with active monitoring meant completion of the asthma diary (a seven-item questionnaire) daily task and completion of at least one set of peak flow readings (three readings per set) per day. Daily compliance for the passive monitoring task meant wearing the smartwatch for at least 12 h between 00:00 and 23:59. The mobile app provided daily notification reminders to complete the monitoring tasks.

The day 0 compliance was the daily compliance on the first day of data collection. The average compliance over each month was calculated across the study population, which was the total tasks completed over 30 days by all participants divided by the total engagement requested. For example, the average compliance of the asthma diary in the first 30 days for 22 participants was

$$\frac{\mathrm{Total\,asthma\,diaries\,completed\,in}\,30\,\mathrm{ days\,by }\,22\,\mathrm{ participants}}{30\times 22}$$
(1)

Change in compliance over time was investigated using linear regression using R, which gave an intercept and gradient per monitoring task. This intercept represented the initial level of compliance, and the gradient represented the average increase or decrease in compliance over 30 days.

Participant retention was defined to be the total number of days between the first and last day of engagement with the study. Participants in phase two of AAMOS-00 each had a potential maximum of 184 days of participation.

2.3 Usability Questionnaires

The AAMOS-00 study exit questionnaire about the acceptability and usability of the system incorporated three validated questionnaires. Usability was assessed with the System Usability Scale (SUS) [12], personal motivation to use technology for self-management used the mHealth Technology Engagement Index (mTEI) [13], and app quality and perceived impact used the User version of Mobile Application Rating Scale (uMARS) [14]. Some uMARS questions were adapted to reflect the AAMOS-00 study’s aims and system.

System Usability Scale (SUS).

The SUS [12] questionnaire is a widely-used 10-item validated questionnaire [15] assessing system usability. It is answered on a five or seven point Likert scale. The questions are simple and effective [15], and alternate between positively and negatively worded text, to reduce response bias [12].

mHealth Technology Engagement Index (mTEI).

The mTEI [13] is a 16-item validated questionnaire which measures a person’s motivation to use telehealth systems by asking about five main areas: autonomy, competence, relatedness, goal attainment, and goal setting. Assessing the correlation to other related measures, the mTEI developers found significant positive correlations [13] with the Psychosocial Impact of Assistive Devices (PIADS) [16], but low correlations to the Technology Acceptance Model (TAM) [17], and SUS [12], suggesting they were distinct measures [13]. We considered all the questions to be helpful in assessing a person’s self-management status and preferences.

User Version of Mobile Application Rating Scale (uMARS).

The uMARS [14] validated questionnaire builds upon the MARS [18] which measures app quality. The questionnaire includes 16 questions in four domains: engagement, functionality, aesthetics, and information, and two sections on subjective quality and perceived impact. The answers include five statements along a scale of “1. Inadequate” to “5. Excellent”. To be consistent with the other two questionnaires (SUS and mTEI), the uMARS questions were reworded to a five-point Likert scale format in the AAMOS-00 exit questionnaire.

3 Results

3.1 Study Population

Most participants in phase two of the AAMOS-00 study were female (77%), white (95%), and had uncontrolled asthma in the month before joining the study (see Table 1). The average age was 40 years, and all the participants in phase two of the AAMOS-00 study had at least 50% compliance in phase one (one month of daily questionnaire completion).

Table 1. Population Characteristics. Twenty-two patients participated in phase two of the AAMOS-00 study, where participants conducted six months of monitoring using smart devices and answered daily questionnaires about asthma. RCP3 score ranges from 0 to 3, 0 indicating good control, 3 indicating poor control [19].
Full size table

3.2 Compliance

The compliance to monitoring did not show significant difference between the ‘passive’ smartwatch and ‘active’ monitoring tasks – all were equally low at <50% by the end of the second month. The highest compliance was to the asthma diary task, which started with 82% compliance on day 0 and continued to have the highest compliance throughout the six months. Compliance to peak flow monitoring was the lowest, beginning at 46% on day 0 and dropping to 16% after six months. The compliance to smartwatch monitoring was in-between the compliance levels of asthma diary and peak flow monitoring in all six months (see Fig. 2).

Furthermore, by investigating the linear fit of the change in compliance over six months, we observed the level of compliance to smartwatch monitoring was between the two active monitoring tasks. Although the asthma diary started with the highest level of compliance, it also had the largest drop in compliance per month (−7.6% compliance per 30 days). In contrast, the peak flow task started with the lowest compliance but also had the lowest drop in compliance per month (−4.9% compliance per 30 days). See Table 2.

Table 2. Linear fit of compliance over 6 months.
Full size table
Fig. 2.
figure 2

Compliance to monitoring in the AAMOS-00 study. Twenty-two asthma patients were asked to complete daily monitoring tasks. Compliance was measured by completion of daily asthma diary, wearing the smartwatch at least 12 h per day, and conducting a set of peak flow measurements per day.

Full size image

3.3 Questionnaire Feedback

More than half of the phase two participants (14 out of 22) filled in the end of study questionnaire. However, this small sample of respondents was skewed towards participants who were very adherent to monitoring, even when compared to the study population (in themselves motivated individuals). Respondents to the final questionnaire had averaged 154 days of participation which was higher than the overall average in phase two (123 days). The two respondents with the lowest retention had five and 38 days of participation. The respondent with five days of participation formally withdrew from the study citing frustration with the technology. Median age of respondents was 47 years old (slightly older than the overall phase two study population) with 71% females (slightly smaller proportion of females compared to the overall phase two study population).

3.4 Questionnaire Score

Median SUS score in the AAMOS-00 study was 61.25, which is slightly below the average SUS score of 68 as measured across 500 studies [20]. The median overall uMARS score was 3.44, which is slightly higher than the average score of 3.26 as measured across 50 mental health and well-being apps on the iTunes store [18].

Investigating the uMARS score further, we could see the lowest scored aspects were engagement (AAMOS-00 median score of 3.20, which is still higher than the iTunes average of 2.68 [18]) and functionality (AAMOS-00 median score of 3.5, which is lower than the iTunes average of 4.01 [18]) (see Fig. 3). The two highest scored aspects of aesthetics (median score of 3.67) and information (median score of 3.75) were higher than the iTunes average of 3.49 and 2.88 respectively [18].

Fig. 3.
figure 3

uMARS score of the AAMOS-00 study’s data collection system. The median uMARS overall score was 3.44.

Full size image

The number of asthma diaries completed gives an approximate measure for the engagement with the study. There was a strong correlation (Pearson’s correlation = 0.56) between SUS score and total asthma diaries completed, which suggests that the usability of the system was a major factor influencing engagement with the study.

In general, there was a weak correlation (Pearson’s correlation = 0.28) between the mTEI score and the total asthma diaries completed, suggesting that motivation to use technology was mostly independent of engagement. However, the autonomy subfactor of the mTEI questionnaire had a moderate correlation (Pearson’s correlation = 0.34) with engagement, indicating users who are motivated by a need to be in control of their own health were more likely to complete asthma diaries.

There was moderate correlation (Pearson’s correlation = 0.32) between the number of asthma diaries completed and the uMARS questionnaire score. In particular, there was a moderate correlation (Pearson’s correlation = 0.43) with the functionality section, and a low correlation (Pearson’s correlation = 0.24) with the information section of the uMARS questionnaire. This suggests that participants who found the research data collection system easy to use and considered that it provided useful and reliable information were more likely to engage with the study.

3.5 Study Feedback

The devices (smartwatch, smart peak flow meter, and smart inhaler) were generally reliable, but some participants encountered issues with different devices during the study. One participant pointed out that the readings of the smart peak flow meter did not match with their mechanical counterpart, a recognized discrepancy that could hinder clinical adoption of the device. Although the hardware designers of the smart inhaler device had tackled some problems with false positives and false negatives, there were still comments about the smart inhaler’s reliability. Additionally, some people encountered missed actuations (false negatives). Some smartwatches also had to be replaced after around five months of use, when the device stopped holding charge or failed to connect to the app via Bluetooth.

“The FindAir app was the most useful tracking inhaler usage but needs to be more reliable. It kept missing uses.”

– participant (female, 38 years old, 183 days of participation)

“The smart peak flow meter was not recording at the same reading as the regular more traditional widely used peak flow tube issued by GP’s and the pharmacy.”

– participant (female, 47 years old, 184 days of participation)

“The peak flow meter did not always work and it became frustrating to use”

– participant (female, 37 years old, 184 days of participation)

“The FindAir app never worked for me; the peak flow meter occasionally didn’t work and the [smartwatch] had to be replaced.”

– participant (female, 46 years old, 184 days of participation)

“I have a iPhone 11 [and Apple] watch 3. Both are far more advanced and could do a better job more accurately and reliably”

– participant (male, 52 years old, 5 days of participation)

Although the smart peak flow meter worked well in controlled environments, it had trouble calibrating with some LED lights when used in a real-world setting in this study. The flickering lights would sometimes drastically inflate the peak expiratory flow (PEF) rates to impossible values. A few participants who could not use the smart peak flow meter reliably used other peak flow measurement methods and manually shared their PEF recordings via email.

“[The peak flow meter] doesn’t work in normal indoor light settings which made it harder to use in autumn and winter when day light hours are limited. … Gave drastically inaccurate readings occasionally.”

– participant (female, 47 years old, 184 days of participation)

“Downside is that the peak flow doesn’t work with LED lighting (lightbulbs we have in UK)”

– participant (female, 48 years old, 184 days of participation)

The AAMOS-00 study was an observational study and did not actively provide any medical advice, but some participants found the monitoring alone to be useful. This included, for example, seeing the disparity between the measured relief inhaler usage and their own answers to the question about daily relief inhaler usage.

“I was surprised by how out I was when guessing how many times I’d used my inhaler.”

– participant (female, 54 years old, 184 days of participation)

“Thank you for sending me [the FindAir] device as it has opened my eyes up to how much stress affects my asthma and is a big trigger for me. It made me realize that I need to be more aware of this and take more action.”

– participant (female, 47 years old, 184 days of participation)

In contrast, some respondents did not think the study and monitoring had changed their attitudes toward improving their asthma.

“I was very happy to record the data, but did not find it helped me to manage my asthma.”

– participant (female, 46 years old, 184 days of participation)

During the study, some participants encountered multiple issues with the technology (e.g. setting up the Bluetooth connection between Mobistudy and the smartwatch). We resolved most software problems via emails and video calls with participants, but some issues were escalated to the Mobistudy (provider of the main system for data collection), Smart Asthma (provider of the smart peak flow meter), and FindAir (provider of the smart inhaler) technical team. Hardware issues were resolved sometimes by detailed instructions or by sending replacements.

“I would like thank Kevin Tsang for his rapid and patient help when devices didn’t work.”

– participant (female, 46 years old, 184 days of participation)

“Thank you for the support when issues did arise.”

– participant (female, 37 years old, 184 days of participation)

4 Discussion

We have found no evidence that a passive monitoring task (wearing the smartwatch) provided a higher level of engagement when compared to active monitoring tasks (completing asthma diaries and taking peak flow measurements) used in current practice of asthma self-management. The compliance to monitoring with the smartwatch was between compliance level to monitoring with the asthma diary and the smart peak flow meter – and both fell off rapidly so that by the end of six months only a quarter of people were still monitoring. This result could be confounded by the technical issues with the devices, because the asthma diary task had minimal technical issues, whereas several participants encountered issues when using the smart monitoring devices.

Feedback from users revealed the challenges with the three monitoring devices, especially with taking peak flow measurements. Although the mobile asthma diary task had fewer technical issues, there was a relatively similar levels of compliance (10%–20% difference) between the daily diary and the daily smart peak flow meter measurements suggesting that the participants were highly motivated to handle technical issues. Overall, the technology would need to be more reliable before it could be widely adopted.

Furthermore, due to the technical implementation of the smartwatch data collection, it required some active engagement from users to upload the smartwatch data weekly to their smartphone which may have been a significant disincentive. This limited our exploration of the potential for fully ‘passive’ monitoring requiring no effort on the part of the user once it has been set up. Technical issues and smartwatch implementation may have led to lower compliance than expected [21, 22]. Moreover, some patients already owned and regularly used a smartwatch, which may have affected their willingness to use a secondary (likely less sophisticated) device for the study.

When compared to other studies and published apps, the AAMOS-00 study’s data collection system was similar in quality, evidenced by standard questionnaires SUS and uMARS. However, the small number of respondents were likely to have been skewed toward highly motivated participants who found the system more usable as they had a higher average retention compared to the overall study population. The usability scores should be interpreted considering this possible bias.

Another limitation was the narrow selection criteria, which selected asthma patients who had an interest in monitoring and had experienced a severe asthma attack in the past 12 months, yielding a small sample size of the AAMOS-00 study. The average retention in phase two (which included daily tasks) was 123 days. This is similar to the average of 122 days patients have been willing to engage in previous studies [7, 22,23,24]. However, it is plausible to expect a substantially lower level of retention in the wider population. During a patient and public involvement focus group session that we undertook, some patients suggested that the retention amongst the wider population could be as low as one week.

There are still areas of unexplored questions within the AAMOS-00 dataset. Our future work includes deeper analysis to investigate each device through correlating themes in user feedback with compliance data. Future studies could consider investigating the effect of reminders and other interventions (e.g. improved feedback and gamification strategies) to increase compliance over an extended time and explore the nuanced barriers of each monitoring task. Additionally, future studies may consider extracting data from the devices patients may already be using.

5 Conclusions

In the AAMOS-00 study, a small-scale study conducted with highly motivated patients, the compliance to passive (smartwatch) and active (daily asthma diary and peak flow measurement that are currently used in asthma self-management) monitoring was similar. Although the AAMOS-00 study faced some technical issues, the quality of the data collection system was comparable to other studies and published apps and is a promising option for future mHealth studies.