1 Introduction

1.1 Challenge of mental disorders

One in four Europeans report that they have suffered from at least one mental disorder during their lifetime [1]. In the USA, 26.2 % of the population, 60 million people, suffer from at least one form of mental disorder [39]. Mental disorders can have a significant, negative impact on sufferers’ lives, as well as on their friends and family. More than 90 % of people who commit suicide have a diagnosable mental disorder [14]. The economic impact is significant as well: Bipolar disorder, one form of mental disorder, is responsible for more disability-adjusted life years than all forms of cancer, each case causing lifetime costs of up to 600,000$ [6].

However, fewer than 40 % of the people with serious mental illnesses receive stable treatment [38, 40]. There are several reasons for people failing to seek treatment (72 %) as well as for dropout (58 %) [38]. By far, the main reasons include people’s wish to solve their problem on their own, as well as situational and financial barriers [38].

Self-assessment and monitoring tools combined with professional decentralized and user-centered pervasive healthcare systems have the potential to reach more patients more efficiently and therefore decrease their suffering [2].

1.2 Smartphones as mobile healthcare sensors

From a technological point of view, mobile phones offer a promising hardware platform for various applications in the pervasive and mobile healthcare domain. In developed countries, almost everybody owns at least one mobile phone; worldwide this adds up to 4.5 billion users [18]. The trend is continuing, reaching out to developing countries and making phones ideal tools for leveraging the vision of providing healthcare everywhere, anytime and to anyone [2].

The modern smartphone is an ideal platform for pervasive healthcare applications. Here, we outline three key reasons. Firstly, smartphone ownership has increased exponentially in the last decade. Smartphones accounted for 51.8 % of mobile phone sales in 2013 worldwide [26]. As of 2014, one-third of the currently worldwide used mobile phones are already smartphones [18]. In contrast to the dissemination of other technologies, the rapid uptake of smart phones is not restricted to developed countries [88]. Their cost has dropped dramatically and functionality continues to expand.

Secondly, a smartphone is a powerful technical platform. It comes with extensive computational power in terms of CPU, memory and battery power; it has extensive communication capabilities with built-in network interfaces for 3G/4G, Wifi and Bluetooth; it is equipped with a number of sensors including accelerometers, GPS, microphone, proximity sensors; it has powerful touch-enabled displays that support advances gesture-based input and visualizations. In many respects, a modern smartphone is a more sophisticated computational platform than a regular desktop computer. This aspect is described in more detail in Sect. 3.

Thirdly, a smartphone is personal and is almost always with the patient [15]. This is especially useful in automated data collection, where the mobile phone acts as a wearable sensor platform located in the user’s pocket. Furthermore, studies have shown that when using a personal mobile phone, the quality of the data collected through questionnaires is of a much higher quality compared to the paper-based self-assessment diaries traditionally used [5]. Using a paper-based diary relies on patients remembering to carry them with them and fill them out, which often results in poor adherence rates, last minute, retrospective completion and memory errors [75]. In contrast, adherence and compliance rates are significantly improved when patients use personal smartphone since a phone is much more easily available [5].

1.3 Smartphones for mental health

Taken together, the above factors make smartphones an ideal platform for supporting personal, pervasive healthcare services to patients. Looking more specifically at supporting mental health, there are more additional benefits to the smartphone.

First, many mental disorders are treated through a combination of pharmacotherapy (i.e., medication) and psychological treatment such as cognitive-behavioral therapy (CBT). The treatment objective is basically to reduce the symptoms of the disease through the use of medication and to help patients identify and change their behavior in a healthy manner. A smartphone-based system can be effective in providing real-time feedback to patients about behavioral patterns and help instruct them how to change them, including through adherence to medication. As such, CBT behavioral training based on multimedia content (text, pictures, videos) can be provided directly to patients over the phone right in situations where it is needed. For example, instead of providing instructions on good sleep hygiene in a clinic during the middle of the day, such instructions can be provided to the patient “right-on-time” before going to sleep. Similarly, continuous feedback on adherence to medication can be provided without the need for the clinic to interfere.

Second, consultation with patients plays a core role in the treatment of mental disorder. Since a smartphone is also a (video)phone, reminders, communication and remote consultations become feasible. For example, in many studies clinicians can send text messages to patients or phone them if needed, and in the treatment of, e.g., schizophrenic patients, video consultations have been applied. As such, the smartphone also becomes a platform for telemedicine in mental health.

Mobile phone usage by our target group, patients with serious mental illnesses, is 72 %. This is a bit below the average usage of the general population; however, non-users have expressed interest in using a phone if beneficial for their health management [7]. Among medical providers, smartphone penetration is already 85 %. Their most commonly used applications are drug guides [23].

1.4 Some examples

In practical studies, patient adherence rates for smartphone interventions are high. McTavish et al. [54] introduced a smartphone application to support patients with alcohol addiction; 16 weeks after downloading the app, more than 70 % of the initial group were still actively using it. Kuhn et al. [43] surveyed perceived helpfulness and satisfaction of post-traumatic stress disorder patients who used a smartphone application as self-management tool for their mental disorder. The results presented were promising but only preliminary. To further increase the acceptance of medical apps, Wac argues important focus points include the reliability of the service and the credibility of the information and the provider [83].

Another recent project called MONARCA focused on bipolar disorder patients and used smartphone sensors as well as additional wearable sensors [68]. The analysis of acceleration data collected with smartphones revealed a correlation between physical activity levels and psychiatric assessment of depression [64]. The practical issues and limitations for integrating this mobile technology into clinical practice are discussed by Luxton et al. [49].

Ehrenreich et al. [16] reviewed eight randomized trials assessing the effect of mobile phones and handheld computers used in psychiatric treatment. The trials targeted smokers who wanted to quit and anxiety patients. The intervention groups had significantly better outcomes than the control groups in the majority of cases.

Another great review of smartphone capabilities and available applications to gather behavioral data is done by Miller [55]. He expects 5 billion smartphone users by 2025 and points out potentials from a medical and technical point of view. He assumes the market of mental apps to grow significantly; however, he does not describe how this process can be initiated and how this influences the requirements on the technical side.

Heron and Smyth [34] reviewed 27 intervention studies in which palmtop computers or mobile phones were used to extend traditional psychosocial and health behavior treatments and therapies. They found that this kind of mobile treatment which is provided to people during their everyday lives can be effectively implemented for a variety of health behaviors and psychological symptoms. Their work focuses on medical results, they omit most technical or legal details.

The most recent systematic review of controlled mobile health intervention studies was performed by Free et al. [24]. They developed evaluation criteria to rate the effectiveness of interventions supported by mobile healthcare technology. However, medical evaluation is only one aspect of research into mobile health care. This paper, while referring to Free et al.’s article, explores the technological, legal and implementational challenges.

1.5 Contribution

This work gives an overview about technical, legal as well as business aspects of mobile health care for mental disorders. This work contributes to the state-of-the-art in the following respects:

  • Legal framework We point out in which direction the regulatory framework is developing, how this influences the requirements on new technologies or applications and name examples.

  • Phones as human–computer interface We suggest design criteria and conclude from related work how interfaces and system architectures should look like to most beneficial support mental disorder treatment in terms of patient adherence, user acceptance and quality of self-assessment data.

  • Phones as sensors We describe technical possibilities which data can be obtained with mobile phones and potentials how to exploit this data in mental health care. Existing work is considered to compare and discuss results and draw conclusions.

  • Business models The success of a new technology is significantly influenced by its compatibility in the market. We point out possible market strategies and its implications on the product development.

1.6 Paper organization

Section 2 looks at the phone as a human–computer interface (HCI), featuring reminders, communication channels to caregivers, and to support self-assessment by patients. Section 3 outlines how mobile phones can be exploited as sensor devices to unveil behavior patterns and voice characteristics of users. In Sect. 4, the implications of the legal regulations governing health research, especially in the area of mental disorders, are discussed. Finally, Sect. 5 presents potential business models that can be used to introduce the proposed technology into the market.

2 Human–computer interface

2.1 Smartphone design features and dimensions

A number of systems for mental health based on smartphones have been designed for depression [9, 37], bipolar disorder [5, 25], borderline personality disorder [69], as well as more general purpose systems for mood charting in CBT [50, 57, 84]. These applications have implemented a range of features, which are targeted at different needs in different mental disorders.

Fig. 1
figure 1

Double-loop treatment setup applying smartphones in the treatment of mental disorders

Full size image

However, looking across these systems an overall system design and a set of core design features crystalizes. Many of the systems implement a common overall system design consisting of a smartphone app, a server for data storage, and a web portal for browser-based access. The setup is illustrated in Fig. 1. Treatment is supported through a double loop: In the patient loop, self-assessment and automatically collected sensor data are sampled by the patient’s smartphone, which is uploaded and processed in the back-end server setup. This data processing provides different data visualizations and other kinds of feedback to the patient on the smartphone. In the parallel clinical loop, clinicians (i.e., physiatrists and nurses) oversee the incoming data and provide therapeutic feedback via the system or engage in direct communication with patients. The core features in this system are self-reporting, automatic data sampling, behavioral patterns recognition, data visualization, therapeutic feedback and communication. Let us consider these in turn while using the design of the MONARCA system [5, 25] and the Mobilyze! system [9] as examples. Both of these research projects use mobile phones for the treatment of depression or bipolar disorder; they incorporate all of the mentioned core features but each comes with different add-ons and different experiences.

2.1.1 Self-reporting

Patient self-reporting is a core to treatment for most mental disorders and plays a key role in, e.g., CBT. Therefore, most of the early work on using smartphones in mental disorder treatment focused on collecting self-reported data on cognitive and behavioral parameters as mood, stress, sleep, activity and coping. Self-reporting is often implemented according to established and clinically validated scoring charts (such as the −3/+3 mood scale used in bipolar disorder and transferred to the MONARCA system, as shown in Fig. 2b), which leads to a simple questionnaire on the smartphone. Taking advantage of the interactive nature of the multi-touch interfaces of a smartphone has been suggested. For example, the mobile therapy app suggested using the touch screen to input mood scored in a special-designed “mood map” [57].

Fig. 2
figure 2

Smartphone–user interfaces of the MONARCA Android application: a Android wall paper with colored “speech bubbles” representing impact factors relevant to the patient; b the self-reporting form; c historical data visualization; d feedback to the patient

Full size image

2.1.2 Automatic data sampling

Utilizing the sensor platform available in modern smartphones offers significant potential for the treatment of mental disorders. Since behavior is such a core factor in mental disorders, the ability to monitor it is crucial. Data on physical behavior in terms of activity and mobility can be sampled through the accelerometer and the location sensors (e.g., GPS). For example, the Mobilyze! system [9] for patients with depression, sampled data from the accelerometer, the GPS, the network interfaces (Bluetooth, Wifi) and ambient light. Besides automatic sampling of physical behavior, the smartphone is also the perfect platform for sampling social behavioral data. It is well known that there is a strong correlation between a mentally ill patient’s social activity and the state of her or his illness. For example, a bipolar patient in a manic phase is by far more talkative than when depressed. By simply monitoring the number of in- and outgoing messages and phone calls, the system can make an accurate sample of such social data, which has been shown to correlate with the mood of the patient [25]. In general, automatic sampling of data is a strong feature of any smartphone-based health technology for mental diseases, since it by nature does not involve the patient and can run unobtrusively in the background.

2.1.3 Behavioral patterns recognition

Higher-level behavioral pattern analysis and activity recognition based on automatically and self-reported data can be of high value in the treatment of mental disorders. First of all, activity recognition can identify specific, higher-level behaviors such as walking, running, sleeping, shopping, attending work. For clinical treatment, this level of activity data is much more useful than, e.g., low-level accelerometer data. For example, the Mobilyze! system [9] was able to recognize the location of the patient and to what degree she or he was with friends. Moreover, by analyzing temporal patterns in data, a system is able to identify patterns and correlations that are not identifiable by the clinician’s sporadic review of data. For example, in the MONARCA system, a patient-specific correlation analysis was able to reveal the relationship between self-reported mood and different so-called impact factors, which were based on the data collected. Figure 2a illustrates how this correlation analysis is visualized on the smartphone applying a “speech bubble” metaphor—each speech bubble denotes the currently most important impact factors for the patient (i.e., the data that correlated strongest with self-reported mood).

Fig. 3
figure 3

Technical possibilities on exploiting mobile phones as sensor devices: a most naturally, voice and SMS data can be obtained and analyzed; b in case of smartphones, additional built-in sensors can increase the data collection possibilities; c external wearable devices can connect wirelessly to the phone and deliver data such as heart rate measurements

Full size image

2.1.4 Data visualization

Different approaches to data visualization have been suggested for smartphone-based health apps. The most basic approach is to simply show raw data visualization applied on a line, pie, bar or similar number chart. For example, in the MONARCA system, data are visualized in a stylized line and a bar chart showing the raw data on mood, sleep, stress, physical activity, social activity, etc. (see Fig. 2c). However, in other smartphone-based health systems, more metaphorical designs are applied. Different more abstract data visualization is a common approach in so-called persuasive applications aiming at encouraging healthy behavior. For example, applying metaphors related to “growing a garden of flowers” [13] or “keeping fishes in a fishbowl healthy” through increased physical activity and exercise [45]. Visualization of data in appropriate charts seems, however, to be the approach taken in the majority of existing mental health applications, such as in the MobileMoodDiary [50], the Mobilyze! [9] and the MONARCA systems [25]. This approach is often feasible since patients are familiar with data visualization from paper-based self-assessment forms. But, the design of proper and useful data visualization metaphors on smartphone-based applications for mental disorders is still an open research challenge.

2.1.5 Therapeutic feedback

Since the treatment of mental disorders is often based on a combination of pharmacotherapy and psychological treatment, the smartphone can become a therapeutic platform. Adherence to medication is core in the treatment of most mental disorders, as it is often a prerequisite for stabilizing the disease. For this reason, some systems incorporate support for medicine prescription (by the psychiatrist) and for the patient to record compliance with the prescriptions. By using the smartphone, medicine can be adjusted in a much more fine-grained manner. By continuously overseeing the disease parameters and compliance to medication, the psychiatrist is able to continuously adjust prescriptions. For example, a psychiatrist observing a patient going from a depressed state to a manic state is able to adjust the medicine accordingly. Similarly, cognitive-behavioral therapeutic feedback to the patient can be delivered on the smartphone platform as small multimedia productions that instruct patients how to handle different situations. Examples include small description of “actions-to-take” when experiencing sleeping problems or when feeling depressed (see Fig. 2d), or as online video instructions, such as available in the myCompass system [32].

Another therapeutic approach for many mental disorders is to support community-based therapy and peer-to-peer support groups. Again, the smartphone is a perfect platform for this. As it embeds a fully functional web browser, it can provide access to many online communities, for instance PatientLikeMe, CureTogether and MyCompass. In such online communities, patients can share experiences and practical advice.

A related, yet quite different approach is to use elements from computer games in the design of smartphone-based health applications. Examples outside mental disorders have been proposed. For example, in the bant system [10] for the treatment of diabetes, users were rewarded for regularly taking glucose measurements with points that could be converted to gift cards in the iTunes Store. Alternatively, users could earn “badges” or compete online with each other. However, in a study of the Fish’n Steps system [45] there was no evidence that gamification had an effect on healthy behavior. It is still an open research question to what degree gamification can play a positive role in the design of personal health technologies for mental disorders.

2.1.6 Communication

Direct patient–clinician communication is clearly a core part of the treatment and care of mentally ill patients. Again, the smartphone is a solid platform for enabling remote, real-time communication using text, images and video. Simple asynchronous messaging has proved to be of great value; for example, studies have reported simple SMS messages, and reminders to schizophrenic patients can have a positive impact on treatment [67]. Smartphone-based systems for mental disorder treatment can therefore fully utilize both asynchronous and synchronous communication. Moreover, communication can be contextualized since communication acts can be tied to events that can be identified in self-reported or automatically sampled data. A recent study [25] implemented rule-based triggers that would show a message to the patient if certain patterns were observed in data, such as not sleeping more than 5 h 3 days in a row. Such automated messages could also be forwarded to clinicians, helping them to identify important patterns in the data.

3 Technical possibilities

The previous chapter focused on using mobile phones as human–computer interfaces for medical applications. However, mobile phones and modern smartphones in particular provide more and more sensing modalities, which enables the collection of valuable data without any additional effort from the user. This chapter focuses on the technical possibilities on exploiting mobile phones as sensor devices. Sensing modalities can be roughly grouped in three classes (Fig. 3).

  1. 1.

    The most natural sensor in a phone is the microphone, which enables voice analysis. Statistical analysis of changes in calling, texting and phone usage patterns allow assumptions to be made about the current mood status (Fig. 3a).

  2. 2.

    Apart from the microphone, an increasing number of additional sensors and communication interfaces are being implemented in smartphones. These include accelerometers, proximity sensors, light sensors, Wifi, Bluetooth and more (Fig. 3b).

  3. 3.

    External wearable devices that communicate with smartphones, usually wirelessly, are another group of sensors. The most common devices are heart rate measurement belts and wrist-worn activity sensors (Fig. 3c).

The technical possibilities within these three groups of sensors are reviewed in the following three subchapters.

3.1 Traditional phones: voice and SMS

Making and receiving phone calls is the original and most intuitive application of the mobile phone. Since 1993 sending short text messages has also become more and more common practice. These two basic functionalities already offer a broad base for medical applications in the area of mental disorders. By listening to another person’s voice or reading written messages, humans are capable of estimating the mental state of a conversational partner. Adapting this intuitive humane analysis to machines is a very active field of research. Analysis of phone calls can be carried out on three levels (Fig. 4): semantic analysis focuses on the content of the spoken words; acoustic analysis extracts features based on the tone and flow of the spoken words rather than the actual content; phone activity analysis is based on the communication and phone usage behavior or patterns. Text messages can be analyzed on a semantic or activity level, a pseudo-acoustic analysis is also possible if emoticons or other non-dictionary symbols are used.

Fig. 4
figure 4

Analysis potential of voice call- and SMS-enabled phones: a semantic analysis focuses on the content of the spoken words; b acoustic analysis extracts features based on the tone and flow; c phone activity analysis is based on the phone usage behavior; d text messages can be analyzed on a semantic or activity level

Full size image

3.1.1 Semantic voice analysis

The first step for analyzing the content of spoken words with artificial intelligence systems is to perform an automatic speech recognition analysis, which transforms the spoken words into text. The next step, the actual content analysis, is based on the assumption that conversation topics differ depending on the medical status of mental disorder patients.

The diversity of expressions used in the spoken word is considered a sign of a person’s creativity. Lauronen et al. found evidence of a link between creativity and mental disorder; however, they have not yet found casual explanations for it [44]. Tager-Flusberg found significant differences in language development of autistic children compared to healthy controls [77]. Autistic children show slowed development rates in most cognitive abilities related to their language. However, the most significant differences are found in their language concerning the semantic and pragmatic functioning. Impairments in these areas persist and are still present in adolescent autistic subjects of average intelligence [4, 74]. A general correlation between psychiatric disordered children with speech and language retardation is also confirmed by Cantwell and Baker [11]. The causality is still controversial. In most cases, the authors assume psychiatric disorders are indirectly caused by speech and language retardation, rather than the other way round.

Computerized voice recognition methods were used by Munoz et al. [61] to detect major depressive episodes. Applied to 104 patients in a clinical setting, the computerized screening method yielded results comparable with those of live interviews in both the English and Spanish language.

To sum up, many studies have proven the existence of mental disorder symptoms in patients’ language; however, the automatic analysis based on semantic voice analysis is still in its infancy. More controlled laboratory studies are necessary before results can be adapted to real-life studies based on smartphone applications. The main two disadvantages from a practical perspective are (1) language dependency, the algorithm has to be trained for the used language and (2) privacy, users might feel uncomfortable knowing that the content of their private phone calls is scanned.

3.1.2 Acoustic voice analysis

To analyze voice on a phonetic or acoustic level, the actual content is not identified. Actually, it does not matter whether the spoken words make sense nor in which language they are spoken.

Fig. 5
figure 5

Acoustic feature extraction: For each voice recording, several thousands of features can be computed

Full size image

For each voice recording, several thousands of acoustic features can be computed [19]. The state-of-the-art procedure is shown in Fig. 5. In a first step, the continuous audio stream is sliced in segments of equal length, for example, 25 ms. For each of these segments, more than 50 low-level descriptors are computed. At the end, low-level descriptors of several segments are merged together to obtain the final feature vector.

Reasonable sizes of feature vectors are around 300–400 features per voice recording [60]. These features are the input for data mining algorithms, like classifications by machine learning or correlations. The ultimate goal is to find dependencies between acoustic features and the diagnosis or severity of mental disorders.

Some acoustic features are known to correlate with the diagnosis of mental disorders. For example, speech pause and speech rate are established measures for identification of depression [76, 79] or mania [89]. They were already introduced before automatic analysis of these features was technologically possible and when mobile phones were still a vision. One of the first studies dating back to 1938 [62] identifies acoustic characteristics of speech that can be observed in specific affective disorders.

Mental disorders often originate from or come together with other chronic and neurodegenerative diseases. The latter are proven to have a direct link to voice disorders, also known as dysphonia [72]. Based on the analysis of acoustic features, Little et al. discriminated Parkinson’s disease patients from healthy controls. They achieved accuracy rates of 91 % [46] and later Tsanas et al. [81] achieved 99 % accuracy for the same task.

Vanello et al. [82] proposed an algorithm to characterize the mood state in bipolar patients. Acoustic features like pitch and jitter are computed after identifying voiced segments in audio files. The authors evaluated their system with a speech database and found significant correlations of acoustic features with patients’ mood states.

Most studies analyze voice in a controlled laboratory environment. Adapting laboratory results and mobile phone recordings to real-life conditions is the subject of ongoing research. One of the first studies involving analysis of acoustic features recorded by mobile phones is presented by Lu et al. [48]. The assessment of the user’s stress level in unconstrained outdoor acoustic environments achieved an accuracy of 76 %.

In a nutshell, acoustic voice analysis is a well-explored field of research. Results of numerous studies have proven the value for classification and identification of mental disorders. Since acoustic voice analysis does not identify the content of the spoken word, it is not language dependent and involves less privacy concerns than semantic voice analysis. In case of shared phones, acoustic speaker-identification can filter out phone usage which is not performed by the specific patient. The main drawbacks are the resource requirements to compute the features, and these can be significant depending on platform and feature selection. As a next step, more experiments have to be adopted from laboratory-controlled environments to real-life settings.

3.1.3 Phone activity analysis

Rather than analyzing the voice in any respect, this method logs data such as the number of phone calls per day, their duration, the time of the day calls are made, the ratio between incoming and outgoing calls as well as the number of unique contacts. Significant changes of phone usage behavior, like an increased number of outgoing phone calls during unusual times of the day, might be a sign of a change in the patient’s mental state. Phone activity is never analyzed in isolation, rather it comes together with voice analysis or is used in conjunction with the analysis of the sensors implemented on the mobile phone (see Sect. 3.2).

The link between social contacts and mental disorders is thoroughly researched. George et al. [27] proved low availability of social support correlates with signs of depression. Another study with 331 subjects suggests a strong correlation of health deterioration with decreasing frequency of social contact [8]. Knutson and Lansing [41] found a link between decreased communication abilities and mental disorders.

Grüenerbl et al. [30] defined phone calls as a social interaction parameter and showed that mildly depressed people exhibit an increased desire to talk, quantified by an increase in length and number of phone calls. Muaremi et al. [60] derived phone call parameters from dyadic phone conversations and showed that the phone call duration and accumulated talking time are the most important variables for predicting bipolar disorder episodes.

In summary, numerous studies suggest that the amount of social interaction of mentally disordered patients correlate with the type and severity of their illness. Additionally, recent studies indicate that statistical analysis of phone activity seems to be a reliable method to quantify a subset of human interactions. However, more studies with more subjects are needed to prove this subset to be representative enough to allow for correct medical conclusions. The main advantage of phone activity analysis compared to the previously mentioned methods is the very low computational effort that the extraction of these statistical features require. Moreover, most of these features do not need to be extracted in real time, instead the processing of the phone log can be postponed to a most convenient time, when the phone is not used and connected to a charger. One disadvantage is the non-availability of speaker identification which leads to biased results in case the phone is shared between different users.

3.1.4 SMS

Most existing studies involving SMS either focus on collecting self-assessment data through text messages or used as a feedback channel from the caregiver to the patient. Moore et al. used SMS to collect self-assessment data. A group of 153 bipolar patients were regularly prompted to complete questionnaires through SMS, and the data were used to calculate mood ratings [56].

Pijnenborg et al. [67] used text messages as a prosthetic aid for compensating cognitive impairments and found it effective for some individuals with schizophrenia. A later study focusing on schizophrenia patients found interventions by text messages to be beneficial for medical adherence and socialization [28]. SMS used as a simple self-assessment tool for bulimia nervosa patients improved their eating disorder significantly within 12 weeks [73]. With similar concepts, text messages also achieved significant reductions of stress, anxiety and depression [32] or to successfully change patients’ behavior [12].

Besides using SMS as an additional communication channel between caregivers and doctors, another possibility is to unobtrusively monitor the natural messaging behavior of patients. This can be done by deploying smartphone applications that do not actively request information from the user, but run quietly in the background.

Similar to phone activity, changes in patient’s behavior can be extrapolated from statistical data from SMS logs, for instance the length of messages, variations in their length, the number of messages, both received and sent, per day. In [70], Rutland et al. introduced measures to analyze the number of sent and received SMS in order to identify addiction.

Beside the log, the semantic analysis of content is performed in the same way as with the content of the speech. Here again, privacy compliance is an important issue. To analyze content of SMS, text scanning and classifying tools can be used. Tauscik and Pennebaker introduced a tool for computerized text analysis methods. This tool classifies words in psychologically meaningful categories [78] and can be used to estimate the author’s mood or emotions.

Today smartphones offer more and more ways to replace traditional SMS technology with internet-based smartphone apps such as Whatsapp, Facebook and Viber. The variety of proprietary systems creates additional challenges for the implementation of message analysis tools. While phone operating systems offer standardized interfaces and methods to access traditional SMS, internet-based communication applications do not follow any standardization, and hence, third-party medical applications cannot access these messages for monitoring and analysis purposes.

To conclude, mobile health applications could potentially benefit from considering text messaging technologies in multiple ways. It could serve (1) as additional form of communication between caregiver and patient, (2) to allow statistical analysis on a patient’s texting behavior and (3) to perform content analysis, possibly complementing the previously mentioned semantic voice analysis. While the first point is already used extensively in numerous studies, for example to obtain self-assessment data from patients or send intervention events to the patients, the scientific benefit of passive usage of SMS such as mentioned in (2) and (3) is still an open research question. From a technical point of view, the automatic processing of text messages can be done more efficient and more reliable than voice analysis during phone calls. However, it is still unclear in which cases the application-specific entropy of mobile texting is comparable to the one of extracted voice features.

3.2 Smartphone-implemented sensors

All smartphones include a number of sensors, which are necessary to provide all the functionalities expected from a smartphone (Fig. 6). For example, there is the feature that the display is rotated and the keyboard enlarged when the smartphone is tilted. This is done by the included 3-axis accelerometer. The availability of all of these sensors (see Table 1) allows entirely new forms of utilization, it can serve as inertial sensor, as GPS tracker or as barometer for sociableness.

Fig. 6
figure 6

Smartphone-implemented sensors are used to obtain data such as location, physical activity and crowd density

Full size image
Table 1 Available internal sensors and their possible medical relevance
Full size table

3.2.1 GPS traces: outdoor location

A GPS is included in the smartphone’s set of sensors so that users can navigate with maps. Storing the respective latitudinal and longitudinal values allows the outdoor behavior of the user to be traced offline. This data can provide a valuable insight into the patient’s state, as from a psychiatric point of view the outdoor behavior of a patient changes according to their mental states [47]. For example, depressive patients limit the radius of action to their flat, or even bed in severe cases, while manic patients feel at home everywhere in the world and might happen to fly to Venice for a coffee.

Possible features that may provide information about the patient’s state and that can be extracted from these GPS traces include as follows:

  • amount of time spent outdoors,

  • clusters (Places where the user/patient spent some time),

  • distances travelled,

  • frequency of visited places,

  • regularity of daily habits.

These features can be analyzed on a daily basis, e.g., by comparing morning and afternoon. However, in order to find correlations between behavior changes and mental state shifts, long-term data series over a series of days or weeks have to be considered.

A study with bipolar disorder patients has confirmed that the amount of time spent outdoors significantly differs between depressive and normal states [30]. Moreover, the radius of action clearly changes with changing states.

From a technical point of view, the main disadvantage of relying on GPS traces is that scanning for GPS reduces the battery life extensively, which may drive the user to turn the GPS reading off. Therefore, it is important to find ways of dealing with data lacks, as was done in [31].

3.2.2 Wifi signal strength: indoor location

Indoors, the patient’s position and location can be determined by the amount and signal strength of visible Wifi stations. Analyses of surrounding Wifi access points help to determine the location of the user/patient when no GPS signal is available, especially indoors. Gathering information about surrounding Wifi access points is completely passive, and no connection has to be established. Potential location accuracy is around 30 m.

Works about indoor location using 802.11 fingerprinting have existed for about 10 years. For example, [66] introduces the powerline positioning system, which uses fingerprinting in order to achieve sub-room-level localization.

In [3], a different approach for using Wifi on mobile phones is presented. The authors introduce SurroundSense, a mobile phone-based system for localization via ambience fingerprinting.

3.2.3 Accelerometer: physical activity

Most modern smartphones include accelerometers. These sensors are necessary to enable the tilting of the screen when the phone is rotated. Nevertheless, accelerometer data includes much more information and can be exploited for a number of additional applications. For example, by subtracting the gravity from the 3-axes amplitude, it can be determined when the smartphone simply lies somewhere or when it is moved. Using this amplitude, a number of other features can be extracted:

  • amount of high/low acceleration,

  • average amount of movement,

  • RMS (root mean square),

  • frequency centroid,

  • frequency fluctuation.

Again, possible changes in the measure of activity over the course of time could be an indication of changes in the well-being and mental state of the patient. Enhanced activity could indicate the beginning of a manic episode, while reduced activity could signal change into a depressive episode.

Different works have already shown that the activity level of bipolar patients is significantly different depending on their mental states (depressive, normal, manic) [30, 64]. Furthermore, Osmani et al. showed that it is possible to analyze the differences of activity levels of psychologically ill patients depending on the time of the day [64]. Other work [35, 87] also successfully measured the physical activity of people suffering from mental illnesses using accelerometer sensors.

3.2.4 Bluetooth device detection: sociableness

Nowadays most cell phones have Bluetooth available to be able to connect to several devices. Most people do not turn Bluetooth off even though it is not used currently [85, 86].

It is possible to use this fact in order to distinguish the density of people in proximity of a cell phone. More specifically, reading of Bluetooth traces should enable one to determine whether a patient is in the presence of (many) other people, or is alone.

In different studies [85, 86], it has been shown that it is possible to attain an accuracy level of 75–80 % of crowd density by simply scanning for visible Bluetooth devices.

3.3 Accessories: wearable devices

Emerging wearable sensor technologies offer the possibility of going out-of-the laboratory and monitoring people in daily life.

There are two different types of sensors built into wearable devices: physical and bio-physiological. The physical sensors (Fig. 7a), mainly accelerometers, are used for activity detection of the upper body or the arm depending on where the sensor is worn. The bio-physiological sensors (Fig. 7b) on the other side measure the biological response of the body such as heart rate, heart rate variability (HRV), pulse, respiration or galvanic skin response (GSR).

Fig. 7
figure 7

Different types of sensors in external wearable devices: a physical sensors are used for tracking of physical activity; b bio-physiological sensors measure biological activity of the human body such as galvanic skin response, respiration or heart rate

Full size image

There are several commercially available devices on the market. Most of them are designed as end-customer products, with the focus on appearance and robustness. Examples for these Activity Tracker & Fitness Monitors are Nike+ FuelBand [63] or FitBit Flex [21] worn on the wrist or Wahoo BlueHR [22] worn on the chest. The wrist band Empatica E3 [17] or the Zephyr BioHarness 3 [80] are mainly used by research institutes interested in enhancing existing biomedical signal processing algorithms.

Typical examples of wearable device applications in the area of mobile health include: stress monitoring in everyday life and the workplace [58, 59]; and in the early detection of Parkinson’s disease [52, 53].

The data of the wearable sensors are usually transferred to the smartphone in real time. This offers the option of predicting a person’s behavior online through the smartphone in a multimodal approach, using both data from the smartphone and wearable sensors. Moreover, transmitting data continuously simplifies its synchronization for offline calculations, which in some cases may otherwise be a difficult task.

4 Legal issues

While conducting clinical trials using ICT solutions, it is necessary to comply with a series of requirements related to the adherence to different policies and regulatory frameworks. The requirements and efforts needed to overcome the compliance with such policies typically vary from one country to another making it difficult to align the planning and execution of cross-countries experimentations in a coordinated way. In the following, we focus on applications using mobile phone technology to deliver health care, so-called mHealth applications. Among the different requirements needed to conduct mHealth clinical trials, the most common are explained in the following section.

4.1 Regulatory requirements

Healthcare solutions using mobile phone applications are not yet governed by a clear regulatory framework at the international level. To this regard, some pioneering work has been done in the USA by the Food and Drug Administration (FDA) who recently released a first specification on the use of mobile medical applications [20]. The FDA made a clear distinction between mobile apps for health and mobile medical apps. The first one refers to applications that can support personal health management without being considered a medical device, while the second one refers to those apps (1) meeting the definition of a medical device, (2) are an accessory to a regulated medical device or (3) transform a mobile platform into a regulated medical device. Some examples of these different categories are included in the specification and mentioned in Table 2.

Table 2 Examples for classification between health apps and medical apps according to [20]
Full size table

4.2 Ethical approvals

User studies conducted by ICT researchers typically need to comply to general rules for information consent, privacy management and data handling. However, conducting multidisciplinary work in the ICT and clinical domain requires more sophisticated set of rules to be satisfied such as clearance of ethical issues. The application for ethical approval can be time-consuming and burdensome for researchers not used to such practices and the associated procedures may differ from country to country and often from one disease domain to another. In particular, applying for ethics consent for trials on research involving the monitoring of behavioral patterns of patients in the field of mental health can require a long time and meticulous procedures raising the risk of delays in experiments. An example of the challenges presented by ethical standard regulation is presented by Mayora et al. [51] within MONARCA project. The three main issues were as follows:

  1. 1.

    The uncertainty in the definition of mobile phone as a medical device (or not); as mentioned above, at the time of this research there was no clear regulation on the definition of mobile phone apps as medical devices; therefore, the ethics committees had trouble giving consent due to the absence of precedents in this regard.

  2. 2.

    The use of non-certified devices for extracting biophysical parameters; as part of the research project, new wearable sensors for monitoring galvanic skin response were developed. However, the use of such sensors was limited to laboratory tests and their use was not authorized for clinical trials given that the devices were early prototypes and not clinically certified.

  3. 3.

    The need to obscure any possible trace of patients identity including locations, personal information, speech and any other possible signal from which identity may be reconstructed also posed a challenge to the project. In compliance with ethical committee recommendations, the strategy MONARCA followed after was to use a privacy-by-design approach. The features extracted on each monitoring modality were coded in a way that did not allow the reconstruction of the original signal. An example of this is the speech signal that was cut into short time intervals (10 ms) and then scrambled. This way it was impossible to reconstruct any content in the speech while physical feature extraction (pitch, intensity, etc.) was still possible [29].

5 Implementation and business models

In terms of a business model, the services provided by psychiatric health monitoring (see e.g., [71], [32, 65]) cannot be thought of in isolation. Together with other types of data contained in the patient’s electronic health record (HER) such as medical summaries, e-prescriptions, laboratory results and radiological studies, the system enables patients and clinicians to

  1. 1.

    track the development of the bipolar disease until the present point (longitudinally, with a view toward episodes and other influential factorsFootnote 1),

  2. 2.

    establish a feedback loop for the patient at the current instance (alarms and warnings as suggested by the system, e.g., concerning early warning signs; clinical feedback sent by clinicians),

  3. 3.

    predict the future development of the disease, based on a trend analysis (subjective: through daily patient questionnaires; objective: through sensor data from the cell phone and other medical devices such as EEG).

Among the stakeholders in this environment, we can now identify likely clients that would benefit directly or indirectly from the system, thus creating value. We are not specifically dealing with legal aspects of that choice, as this differs between countries, medical bodies and even clinical departments. However, this does not free us of ethical correctness, meaning that the well-being and dignity of patients are central and must be respected at all times.

5.1 Clients

Patients (more correctly called clients in that context) are the core users, but not necessarily the key payers. We have 2.6 % prevalence in adults ([39], Table 1), which might seem quite a large population. If there was only one storage for this data (i.e., a single psychiatric EHR domain), then provision could indeed be quite cheap. However, this is unrealistic as the data originate in different counties, in which there are several regional health providers (health associations running their clinics) and sub-providers (e.g., laboratories) with a variety of document sources (HIS, DIS or RIS, to name just a few) and different legal obligations (cross-departmental rules in a clinic; cross-provider rules; cross-county rules). Thus, in reality, such work requires a fairly large amount of coordination and interfacing, in a standardized (e.g., HL7, IHE) or proprietary form (e.g., XML) and the costs are usually high. Standards may require lots of work to be natively implemented in the software tools where the data originates, whereas proprietary solutions that act as adaptors are hard to maintain. Legal issues between data providers are another aspect to be paid for. In short, costs are generally too high for a single patient to be able to afford the service, and thus, some form of cross-financing is needed.

Giving access to caretakers is another possibility for adding benefits to both the affected person as well as his/her family and friends. Bipolar patients typically have a custodian before, during and after being hospitalized. Apart from the patient’s well-being, this person is responsible for maintaining financial affairs. Offering access to this group (based on a patient’s wish, of course) may be beneficial, so that both current status and prediction can be incorporated into the supervision. Both near relatives as well as appointed custodians can take this role. However, from a business standpoint, relatives may indeed be willing to pay for this service, while appointed custodians would need to be re-financed through the medical system. The first case might furthermore be legally tricky, since access to the documentation would actually be required for much longer than the actual hospitalization—it might be that the family wants to know what happens with the patient even though the disease is not in one of its severe phases.

Clinicians (or rather psychiatrists, in this case) are responsible for the patient, and thus require access to the psychiatric health record as a tool for treatment. However, the benefits cannot be found in the (intramural) treatment process, but rather in the time spent outside the hospital. Thus, argumentation needs to focus on indirect returns. Naturally, this needs to be considered at a higher level than just an individual clinic. If it is true that we can decrease treatment costs while at the same time increasing the well-being of the patient extramurally, part of the saved costs can be reallocated for giving access to the system. However, we would note that the clinical process has to be changed as well. Instead of a treatment that would focus on a pipeline in which the doctor acts an end point, we would argue for a bidirectional communication pipeline supported by our system. Of course, this comes at a cost. Automated analysis features can give warnings to doctors and patients when a significant status change occurs, but these will in turn raise workload in the treatment process. All in all, we feel that it is more about redirecting now-common clinical costs in the treatment process for the sake of treating the patient extramurally. This can raise some legal issues with looking at the treatment which now becomes more of a regular task instead of one that is clearly subsumable under a medical case. The client in that case is not the clinician, but the medical body to which he belongs.

In continuation of the medical body being the client, we can also say that the psychiatric record as a service, augmented with data from automatic analysis of sensor information and questionnaires, can be included into a national (or one day may be cross-national) medical record. One example for the latter would be the Austrian ELGA, which is operated centrally by the main social insurance but fed by each individual county. The client in that case would be the medical system itself. On the top of the technical infrastructure, we have the national patient index. Paving the way for psychiatric data to enter the central domain is already a business opportunity, and providing the necessary level of security and operating the connection to the central system is a project of its own. Vice versa, connecting individual clinics with their psychiatric data is also an opportunity; however, this requires a strong integration with the HIS (and is thus best suited for HIS providers).

Third parties, such as insurance companies, were not considered by us, as we feel that their role in the actual treatment process is very limited. Nevertheless, some features, such as the analysis of early warning signs, might be applicable for prevention of other mental diseases and may thus be used beyond bipolar disorders. However, this remains speculative, and outside of our scope.

5.2 Possible business models

Among several possibilities for modeling the revenue chain for a psychiatric monitoring system (see e.g., [42]), we now discuss two complementary options often used in the provision of software:

  • Freemium (free/premium, [33]) strategies might be used in cases where a base service is augmented by extra functionality. A classical example is the longitudinal (psychiatric) health record, which is presented as is, with documentation ordered by time (base service). For an extended fee, these pieces of data could be given visually, by composing graphs and charts showing the data down to the level of its contents (e.g., fields contained in a laboratory report). Likewise, search functionality and personalized views according to favorite filter criteria could be a sales opportunity for both the clinical environment as well as the patients. For clinics, faster access to the data (e.g., through full-text indexing, pre-fetching, high-speed data connections) might be another important factor affecting work efficiency with the system, thus also becoming factors in premium-level access.

  • Subscriptions may be used for actually implementing the freemium model, if the costs are to be distributed over a longer span, thus making it more likely for patients to use extended features. The base subscription for the psychiatric record service is implicit, generating a constant revenue stream that is needed for the initial implementation, operational costs and maintenance.

A concrete example for both cases is the health monitoring service dacadoo (www.dacadoo.com), which is free for all persons under 20 and then becomes a subscription service (currently 5€ per month). In the psychiatric domain, personalized portals such as Optimism (www. findingoptimism.com) offer a dual concept, free for clients, with a monthly subscription fee (39$) per clinician. Interestingly, the border between monitoring applications and the used hardware (i.e., sensors) is becoming quite elusive; for example, Philips DirectLife (www.directlife. philips.com) offers an activity sensor and at the same time a monitoring portal for providing feedback. Segregation into different sensor-centric domains is not a problem so long as data integration into an aggregated health record happens in the background. This is yet another business opportunity.

5.3 The big picture

Putting together the presented context, clients and business models, we could think of a base service being financed by the medical system (national health insurance, health provider, clinic), in which an EHR, HIS or other document source is extended by the base psychiatric monitoring system in the following fashion (also refer to Fig. 8). Importing sensory data from a mobile phone would lead first to basic processing and then higher-level analysis, and then to a current patient status that is made available to both clinician and patient as documents. Depending on the presented fields of application, we may now extend this and distinguish by:

  • Past The provided documents, alongside existing documentation, offer a view of the longitudinal development of the disease. More insight into the history may be given by a subscription service, in which patients and medical institutions pay to see a graphical time line, labeled by episode. Clinics may also be given access to more elaborate tools for mining the patient history, such as in-depth graphs that are based on the document contents. Part of the costs for that can be diverted from savings in the treatment process (i.e., days spent intramurally). That means that the money is provided by the next higher level; for example, the social insurance could finance the medical institution rather than the clinic directly.

  • Present The present status, including immediate feedback (alarms, warnings) to client and clinician, is covered by the base service. However, bidirectional feedback between clinicians and their patients is a more advanced functionality which could be available at a surcharge. As a side note, the time that clinicians spend with the system is outside our consideration, it would likely be charged as documentation during medical treatment. Likewise, communication support between the patient and his/her caretakers is another premium functionality. Patients can give their caretaking agreement in an easier (digitized) fashion using a web solution, and caretakers would be able to track the well-being of their patients in parallel to their development.

  • Future Predictive capabilities are central to such a system’s utility. Trend prediction lies at the core of the services provided; however, the supplier may offer extended capabilities (i.e., longer time spans, upper/central/lower bound for prediction) as a premium service. For the clinical world, future extensions such as validation (predicted state vs. state assessed by the medical staff) may also enter the premium package.

We argue that all parts of the monitoring system should be subject to prescription: The sensors (e.g., activity), cell phone apps (e.g., sleep monitor, mood charts), personalized health portals along with the required technical infrastructure in the background all contribute to the greater goal of sustaining the mental health of the patient. However, argumentation for financial support in the context of mental disease prevention is still in its infancy. Some results clearly point out the benefits even when applied over a short time span [5, 36]. Longer trials and more data on which monitoring and analysis configurations are most effective are yet to come.

Fig. 8
figure 8

Potential solution for system implementation

Full size image

6 Conclusion

Mental disorders are a challenge in the developed as well as developing world, which is forecasted to continue rising. As such, developing effective and efficient treatment is important. The concept of pervasive health care is recognized as one potential treatment option, and smartphones are identified as a promising technology platform through which to implement it. They are already widely available and can serve as human–computer interface as well as sensor device for monitoring the mental disorders at the same time.

Different approaches to implementing HCI on mobile phones for mentally disordered patients have been reviewed, and based on previous studies, suggestions are given. User acceptance and user adherence are very important parameters that can be influenced significantly by the design of the HCI. Moreover, not only the quantity and reliability but also the quality of data is improved when self-assessment questionnaires are issued through thoroughly designed mobile phone interfaces compared to the state-of-the-art paper-based versions.

From a technical point of view, smartphones are becoming more and more powerful and are able to implement more and more features out of the box. Broad data collection is possible, not only based on traditional sensors like microphones, but also including additional sensors like accelerometers, GPS, proximity and light sensors as well as external sensors like heart rate measurement belts. Different analysis methods are reviewed, and results are discussed. The ideal approach strongly depends on the specific application. We present advantages and drawbacks of the most promising technologies with a particular focus on detecting mood disorders like depression or bipolar disorder.

Finally, the paper discusses practical implementation questions regarding legal issues and business model options for introducing successful research into the market. Legal requirements differ from country to country. As a general guideline, the differentiation between health apps and medical apps is most probably going to be implemented under many different legislative systems. An application being classified as medical app has to fulfill similar requirements as traditional medical hardware like infusion pumps or ECG measurement devices, including labor- and cost-intensive certification processes.

Considering the promising results mobile phones can offer for the treatment of mental disorders, commercial healthcare providers are becoming increasingly interested in adapting research into products and introducing them into the market. We discuss benefits that can be provided to patients, caretakers, clinicians and healthcare insurance providers. Furthermore, we identify business models through which these benefits can be converted into payments, making the investments payoff.

We are convinced that mobile phones are beneficial for providing better treatment to more users with less cost, in particular in the area of mental disorders. The main challenges are primarily not technological, rather they lie in the complexity of the healthcare sector. The paradigm shift necessary to implement pervasive healthcare including from mobile phones requires a close collaboration between politicians, researchers and medical staff as well as commercial healthcare institutions. Established infrastructure needs to change to be ready for these advances.