Keywords

The Growing Prevalence of Sensing Technology

Sensing technology has been driven by advances in the semi-conductor industry and has become a mainstay of modern life. Modern sensors are small, affordable and have powerful computing built-in. This allows development of devices that are not simply for data collection but can process data onboard. It opens up options for sensors transforming data before transmission for example to enhance privacy and improve context. Figure 5.1 illustrates a well-established and steadily growing market for personal, wearable devices with sensors built-in, colloquially termed wearables—driven in particular by smart watches and specialist devices for health and fitness.

Fig. 5.1
figure 1

Wearables market over the last 5 years [4]

Full size image

Another trend enabling the hybrid healthcare environment is the rise of spaces and devices with sensing built-in. Ambient or environmental sensors such as location and cameras can combine with sensors residing in objects such as smart speakers, lightbulbs, thermostats, robot vacuums and more to both enable data collection and provision of contextual, personalised environments. Smart-homes [1] have long been studied in research environments with healthcare as a target application. This includes projects such as MavHome [2] and MIT’s PlaceLab [3]. The home may also facilitate the operation of devices in the house based on sensed information. The sensors utilised in these projects include temperature, water flow and utility usage sensors as well as pressure sensors on furniture, proximity sensors for tracking user position in rooms as well as devices for monitoring vital signs. These technologies are now available through commercial technology providers including Google Nest™, Amazon Alexa™ and Samsung SmartThings™, which have created eco-systems for smart devices.

Wearable and ambient sensing can provide detailed pictures of many aspects of human life. Table 5.1 lists some of the commonly used sensors in commercial and research settings, along with an indication of their frequency of use.

Table 5.1 Commonly used sensors with potential for use in hybrid healthcare
Full size table

Unsurprisingly, the potential of these technologies is maximised when leveraging more of them [5]. Ambient sensors impose the least burden on users once in-place—whereas wearables often offer the highest fidelity information. Multiple, independent sources of information increase not just the quantity of information but also the reliability of it, the resilience in the face of device failure and the ability of intelligent algorithms to make inferences from the data. An enabling technology for connecting multiple sensors is extensible, open-standards for connectivity setting up the potential for the ‘Internet-of-Things’ (IoT). These can emphasise low-power, local networking as in the ZigBee standard [6] or can allow ubiquitous wearable and ambient devices access to high-speed internet with the emerging 5G standard [7].

Challenges for Sensing in the Healthcare Domain

One of the most important areas for sensing is healthcare and as a result it has been an area of significant research focus. Chapter 7 outlines in more detail the factors driving this—including the growing healthcare needs of ageing demographics, reducing hospital admissions and the potential for patient centered care and the opportunity to provide better quality care with lower cost. Successful interventions in healthcare leveraging sensors must however overcome serious challenges.

A primary challenge for wearables and remote monitoring is incorporating privacy. The nature of the data imposes a significant cost if privacy is not taken seriously. In conventional healthcare there are stringent standards for storing and transmitting patient data—these must be adhered to [8]. Extensive meta-data about users can be valuable for analytical purposes—however each data source imposes regulatory constraints varying by region. Healthcare frameworks leveraging sensing [9, 10] have proposed addressing this through high security data transmission, leveraging anonymisation when possible, customising software architectures to regional and national regulations, adopting analysis that preserves privacy, and letting users control what information is collected and for what purpose—with conservative defaults.

To move hybrid healthcare into the mainstream it must cater for a wide range of healthcare needs and lifestyles. Individuals with chronic, non-communicable diseases (NCDs) such as heart disease will require different sensing modality and device type compared to women seeking fertility treatment. Lifestyle context—in addition to the underlying data requirements need to be factored into the system design and analysis [11]. The varied contexts for sensors pose challenges to algorithm development as well—it is often difficult to capture these in lab environments where sensors are calibrated and algorithms developed. Adaptive learning algorithms, along with the ability to re-calibrate sensors based on environment are key to overcoming this challenge [12].

Hybrid healthcare must also economically scale to potentially millions of users to deliver on its potential. Systems for hybrid healthcare must be designed with scalability in mind—from how data is transmitted and stored, to how it is visualised [9]. The rapid gains in computing power for small devices offers the potential of processing data on-board. A useful idea here is incremental processing: a substantial gain in performance can be achieved by processing the data as it arrives on devices and transmitting the information derived [13]. This is typically lower in volume and the data would otherwise require server-side computation. This can also be a privacy preserving strategy—for example in the case of vision sensors [14]. Avoiding the transmission, server-side computation and storage of data also has an unexpected environmental benefit—the internet is responsible for an estimated 3.7% of global carbon emissions [15] partially through the need to transmit and store huge amounts of data.

The rapidly standardised technology stack for cloud computing serves as a good foundation for flexible software architectures that can handle information from sensing devices scalably. Services for streaming data [16] for example can process data as it arrives and minimise the need to process at query time. Responsiveness can be enabled in these systems by leveraging events—allowing sensor devices to publish and subscribe to events that influence their behaviour [17]. Commercial standards such as SmartThings™ and IFTT™ make available these technologies for processing sensor data to consumers without requiring deep technical knowledge. To make sensing further accessible it must be available to end-users on personal devices such as smartphones and TVs—where powerful user interfaces leveraging chatbots and augmented reality can accessibly offer users context and explanation about their information [18].

Hybrid healthcare systems must not become the repositories of unutilised and uninformative personal information. Many sources of data will not offer useful information most of the time. This can be filtered at source—for example in naturally event-based sensors such as PIR. Alternatively, when information is fused, algorithms may analyse which dimensions of data offer the most information and choose to keep this at the highest resolution while discarding or archiving useless data. Dimensionality reduction techniques [19] such as Principal Components analysis (PCA) and Manifold embedding can represent information from very many sources in just a few dimensions. Alternatively—feature selection [20] can be used to identify the information most relevant to the context of the user’s activity and only transmit and store this [21]. A further tool available to reduce storage and compute costs is compression and synopsis data structures [22]. Strategies can include downsampling, representation of statistical summaries and leveraging more advanced techniques like wavelet filtering [23] to preserve only the signal relevant to healthcare—not the noise of everyday life.

Applications Enabled by Wearable and Remote Monitoring in Healthcare

Analysing Activity and Behaviour to Improve Care Provision

Activity and behaviour are important clinical indicators of well-being. Activities of Daily Living (ADL) [24] can be used to infer the functional ability of people to live independently. Changes in frequency and performance of ADL may also signal the onset of disease, worsening of a condition [25] or be used to track recovery from surgery [26]. Recognising activities automatically is therefore an important and wide-ranging problem in the field of sensing [27]. A large range of sensors have been used for detecting and classifying activity—including cameras [28], wearables leveraging accelerometers and gyroscopes [29], activity information derived from smartphones [30] and combining ambient and wearable sensing [14].

The varied range of sensing technology offers the ability to customise applications to lifestyles, privacy priorities and health needs. For example, wearable sensors are very effective at differentiating between activities of different physical intensity. Sensors placed on objects can offer location and contextual information without the privacy issues attendant on more powerful, but potentially intrusive sensors like cameras. For example—sensors placed in drug dispensing boxes [31] can help track medication compliance. Cameras can offer the deepest degree of information about the user state—and depending on placement context and how information is extracted can be made palatable to end-users [32]. Infrared sensors offer an intermediate degree of information—these can track the movement of users across rooms. This can be used to generate a picture of an individual’s room occupancy over the day, recognise activities from context (e.g. kitchen or toilet activity) and identify anomalous behaviour [9].

Activity analysis can be divided into two broad tasks—the first, termed ‘activity recognition’ deals with detecting the current user state in a typically small-time horizon. Building on this by incorporating temporal elements or sequencing is referred to as ‘activity modelling’. To some extent, the latter can also be used for modelling complex activities from simpler, ‘atomic’ activities e.g., inferring ‘food preparation’ from a sequence of movement activities and location events. Whereas activity detection typically relies on discriminative machine learning algorithms, activity modelling often leverages sequence modelling algorithms such as Hidden Markov Models [33] and Long-Short Memory Neural Networks [34].

The effectiveness of wearables to identify ADL, as well as the potential to minimise the information extracted from the wearables is demonstrated in [35]. Figure 5.2 shows the data for several activities visualised in 2-D space using the feature selection algorithm Simba [36] and PCA. Both algorithms compress a large number of features extracted from the data in two dimensions, while still allowing the discrimination between impaired subjects (red) from normal subjects (blue).

Fig. 5.2
figure 2

Activities classified from sensor data in low-dimensional representation—blue points are from normal subjects, while red points are from impaired subjects [35]

Full size image

ADLs and how they are performed can also offer insights around recovery from surgery. Current medical practice to assess recovery from surgery relies on patient questionnaires—for example the KOOS questionnaire for knee and mobility [37] and the CHAMPS physical activity questionnaire [38]. These questionnaires impose a compliance burden on patients, are difficult to control for subjectivity and there can be a significant delay between the onset of complications to the filling of questionnaires and subsequent analysis of it. Using sensors and remote monitoring allows for objective measures of recovery and timely interventions based on events automatically extracted from the data. Figure 5.3, taken from [35], shows the discrimination between subjects in early recovery (1–6 weeks after surgery) from subjects in late recovery (12–24 weeks after surgery) using accelerometer data collected while performing Step-up transitions (e.g. stepping onto stairs) and Stand-to-Sit transition (e.g. sitting into a chair). These activities are automatically identified in a continuous stream of sensor data by noting changes in data transformed into reduced dimensions using manifold embedding. The potential for this is to identify patients who do not follow the normal trajectory of recovery and direct care providers to them.

Fig. 5.3
figure 3

Transitions from patients recovering from knee replacement surgery. Figures show embedding of accelerometer data while patients performed Step-up transition (a) and Stand-to-Sit transition (b) [35]

Full size image

When wearable and ambient sensors are used to provide care to patients with chronic illnesses and requiring long-term care, an important aspect of behaviour to analyse is routine. Deviation from circadian rhythms in people can indicate a change in the state of health [39]. Furthermore, key indicators of wellbeing can be observed in daily routines for example sleeping habits, regular eating, exercise levels, routine social interactions. Typically, for this the sensors detecting movement and location both indoors and outdoors are important. As data accumulates rapidly over the longer-term, abstracted pictures of an individual’s routine allow the concise representation of their typical behaviour and allow care-providers to more easily see deviations. An example for this comes from the SAPHE project [40]—which deployed smart-home technology in partnership with Liverpool Primary Care Trust. A range of sensors were used—including wearable sensors such as accelerometers, ambient sensors including PIRs and connected devices such as smart weight scales. The patient cohort was typically elderly and requiring long-term care from community care providers who would otherwise prioritise their work using ad hoc means. Figure 5.4 shows the data abstracted about the individual’s routine behaviour and showing a change in their routine resulting from an intervention from the healthcare worker, taken from [35]. These changes in routine behaviour derived from accelerometer and location sensors can allow timely interventions from care providers.

Fig. 5.4
figure 4

Changes in routine behaviour (in red) compared to normal behaviour (blue) observed in a patient in a smart-home environment. Routine information is extracted from sensor data and visualised in a reduced manifold space [35]

Full size image

Personalised and Preventative Medicine Through Sensing and Smartphones

Widely adopted consumer devices such as smartphones and smartwatches carry a wide range of sensors. Accelerometers, GPS and gyroscopes are almost universal, and physiological sensors such as heart rate and skin temperature are also not uncommon. These devices therefore offer immense potential for enabling personalised and preventative healthcare.

Relatively soon after sensors became crucial for powering smartphone features like managing the screen orientation researchers were interested in studying the potential for use for activity and routine behaviour analysis [30]. Smartphones pose particular challenges however—they may not always be on the user’s person and orientation and placement of the phone is often inconsistent. These factors are addressed—to an extent—in the ActiveMiles project, where rotation invariant features are derived and then clustered to get a basic indication of the user’s activity levels. Combining with location data from GPS can give a rich picture of the user’s activities and behaviour over time. Alternatively, the app can be calibrated with the phones placement at fixed locations e.g. belt buckles or pockets and machine learning models can discriminate between different activities using the sensor data (Fig. 5.5).

Fig. 5.5
figure 5

Features from sensor data visualised in low-dimensional manifold space. Data is collected from a belt worn phone and activities can be clearly separated [35]

Full size image

Wearables such as smart-watches and fitness trackers do not have the same challenges as with data collected from smartphones. Furthermore, a wide range of sensors are available as wearables—activity measures being the most common. These are widely used in the fitness tracking and sports markets and have extensive app eco-systems. More specialised sensors can be integrated into wearable and minimally invasive sensors. An example of this are sensors designed to track period cycles in women. Devices such as the OvuSense sensor [41] measure Core Body Temperature (CBT), changes in which are strongly associated with the phases of the reproductive cycle. OvuSense can characterise typical and atypical fertility cycles in addition to helping predict period and ovulation days [42]. OvuSense is one of the sensors supported by the Nabta app and will be a part of an ecosystem of sensors that Nabta app can draw information from, in addition to logs added by users themselves that give context to sensor data including mood, medications taken, food and water intake and more (Fig. 5.6).

Fig. 5.6
figure 6

The OvuSense sensor designed to track menstrual cycle using Core Body Temperature (a) and a screen designed to convey information extracted from the sensor in the Nabta app (b)

Full size image

Conclusion

Wearable and ambient sensors are inexpensive, ubiquitous and increasingly powerful and easier to connect. A diverse set of healthcare and lifestyle monitoring use-cases are enabled by these devices ranging from care of chronically ill patients and post-operative recovery to delivering personalised medicine and helping patients take control of their health through insights derived from their data. Challenges remain—the need for privacy, catering for lifestyle differences, increasing regulatory burdens and dealing with scale of data and more. Flexible, cloud-base software architectures, judicious data processing and use of compression and leveraging AI and machine learning can help address these challenges and deliver on the promise of sensing in healthcare.