Abstract
This paper intends to decode depressive disorder using computer vision. Facial expressions rendered by a depressive and non-depressive person were studied against a given stimulus. A survey was conducted using Attention Deficit Hyperactivity Disorder (ADHD) questionnaire on a group of four hundred one volunteers between age group of nineteen to twenty-three years. A total of 254 male and 147 female volunteers participated in survey. Three hundred and eighty-seven responses were actually received and seventy-two respondents were identified as potential patients of depressive disorder. Amongst these anticipated depressive patients, thirty-eight were called for a personal assessment/ interview by practicing psychologists as per DSM-V standards. Data collected from these respondents were used to train a Convolutional Neural Network model, so as to classify a person as depressed or not depressed. The proposed system attained a precision of 74 in the identification of depressive patients. This study concludes to the fact that facial expressions rendered by a patient suffering from the depressive disorder are different from that of non-depressive person against any given psychological stimulus. Further, it was also concluded that facial expressions rendered by a respondent against any annotated quality stimulus like ADFES dataset could provide results comparable to that of ADHD questionnaire. The outcome of this research intends to facilitate doctors to identify potential depressive patients and make an early diagnosis.
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Survey of Health, Ageing, and Retirement in Europe have reported a substantial rise in the count of aging depressive adults. Further, a significantly high risk of suicidal behaviour was said to be associated with depression. The dynamic global economy and intense competition for employment opportunities have resulted in a considerable rise in the display of depressive symptomatology. Anxiety and depression have affected nearly twenty present of the world population [1, 2]. Anxiety and depression being internalizing disorders can even impair the socio-emotional development of an individual [3, 4]. Such internalizing factors may lead to health and social problems like psychopathology, suicides, drug abuse and even functional impairment [5, 6, 9, 11]. A depressive disorder is also integral to many impairments related to social functioning, which may persist even during remission [12, 13, 20]. Depression is a leading cause of ill health and disability worldwide. As per the World Health Organization (WHO), depression would be the second leading causes of disability and death in humans in 2020. Further, missed or misdiagnosis of depression may subsequently result in a high recurrence rate [14]. Contemporary diagnosis is primarily dependent on self -report or clinician’s rating scales. Self-report scales and inventories are highly vulnerable to subjective factors of reporting individual and are insufficient to support the diagnosis of depression [15, 16]. The clinician’s rating scales require high-end clinical skills, professional knowledge, and training. Many researchers have related depression with medical domains like neuroanatomy, endocrinology, and physiology [2, 17]. Although there is sufficient literature exploring neuroanatomy, neuroendocrinology, and neurophysiology of depression, yet researchers have not cited any robust clinical test for it. There is no laboratory test available so far, which could be used as a diagnostic tool for this disease. The available laboratory tests sans required sensitivity and precision, hence are not confirmatory [15]. This situation warrants a need for evaluation methods for not only diagnosing depression but also to estimate its severity. To comprehend the underlying mechanisms resulting in disruption of social behaviour of depressive patients, researchers have started understanding social cognition. Social cognition relates to the metacognitive abilities of human beings, necessary to decode mental states [10, 18, 19]. Mastering such mental states may help individuals to understand mutual thoughts, feelings, and intentions [21]. Research suggests that fifty-five percent of human communication happens through expressions, while just seven percent of information is transmitted through language [22]. Expressions are the language of emotions, face, eyes, and hand gestures are their communication channels. Study of interrelated expressions and state of eyes may substantially reveal hidden, yet observable external manifestations of depressive episodes. Such manifestations are actually observed by experienced medical practitioners during personal contact with patients of depressive disorder. The Diagnostic and Statistical Manual of Mental Disorders (DSM-5) recommends the study of facial expressions and human behaviour to infer depression [17]. This observation concludes the fact that there are direct observable differences between the behaviour and facial expressions of depressive patients and non-depressive people. Affective computing has significantly contributed to the understanding of human behaviour, thinking pattern, and decision making. Affective computing is an amalgamative study of facial expressions, posture analysis, speech recognition, along with understanding the correlation between emotion and expression. Depression analysis through facial encoding and emotive analysis has recently gained the attention of the research community. Numerous studies have explored the possibility to understand and identify depressive disorder through visual cues. Till now there is no computer model trained using facial cues rendered by patients of depressive disorder. Depression analysis through facial encoding and emotive analysis has recently gained the attention of the research community. This paper intends to explore the feasibility to differentiate facial expressions rendered by a patient suffering from depressive disorder from that of a non-depressive person against any given stimulus. The outcome of this research intends to facilitate doctors to identify potential depressive patients and make an early diagnosis. Rest of this paper is organized as follows; Section 2 corresponds to the review of the literature and assumed hypothesis. Section 3 details the experiment design. Section 4 describes the case analysis followed by results and discussion in Section 5.
2 Related work
This section comprises of relevant theories, methods, and techniques proposing the fusion of depressive disorder and information technology. First part briefs the proposed assistive technologies for early identification of depressive or psychiatric disorders. Significant platforms, frameworks, and systems for identifying depressive disorder signature shall comprise the second part. Numerous attempts have been made to predict and explain facial patterns concerning the behaviour displayed by patients suffering from a depressive disorder. The proposed theories were based upon emotional and behavioural dimensions and needed a testable hypothesis based on facial structure. This was attained by linking the contractions of facial muscles concerning expressing emotions [23]. Subsequently, many emotional dimensions were explored [24,25,26, 111]. Mood Facilitation hypothesis was proposed to understand the association between moods and emotions. It states that moods would increase the likelihood and intensity of matching emotions, while it would decrease the probability and severity of opposing emotions. As depression is pervasively negative mood is a prime symptom of depression, this hypothesis assumes that a depressed mental state will experience facial expressions with negative valence [27]. Technological advancements have greatly affected established practices of clinical psychology and psychiatry. There are numerous cyber resources imparting knowledge on psychiatric conditions, assessment procedures, and diagnosis [28, 29, 31]. Recently, the research community has explored the use of smartphones for collecting sentiment data [32]. It has increased the reach and propagation of therapeutic assistance for patients with depressive disorder. Many smartphone apps dealing with mental health concerns, such as depression and stress, are freely available over the internet [33, 34]. Internet-delivered treatments are collectively known as Internet-delivered Cognitive Behaviour Therapy (ICBT) [35]. Different versions of ICBT are driven by a software platform, which integrates assessment instruments, treatment materials, and technologies to facilitate early diagnosis of depression [36]. Such software platforms require regular administration to monitor progress, the severity of symptoms, and anticipating the risk of self-harm [37]. Security of patient data is also a crucial concern for such software platforms [38]. A primary challenge for translational clinical psychology and psychiatry is that it cannot deploy machine learning paradigms for diagnosis, prognosis, treatment prediction, detection and monitoring of potential biomarkers of cognitive disorders [39, 75]. Machine learning was subsequently explored to enhance computer-aided psychotherapy [40]. Accurate predictions are attainable through any quantitative data. There are a few assumptions like normality, homogeneity of variance as the estimates of model performance are empirically determined. Machine learning techniques are designed for the multivariate analysis of data sets with high dimensionality even when the ratio of cases to variables is limited [41].
Early machine learning studies studied the fact if diagnostic divisions between individuals could be summarized using high-dimensional data like structural and functional neuroimaging [30, 42]. Early researchers have tried deploying machine learning on mental ailments like Alzheimer’s disease, depression, and Schizophrenia [43,44,45,46,47]. Recently, machine learning has been extended to a broader diagnostic spectrum ranging from an anxiety disorder, drug addiction, anorexia and other phobias [48,49,50, 69]. Literature suggests that machine learning can identify patients with psychiatric disorders with an accuracy of 75% [42, 51, 75]. Researchers have also successfully deployed machine learning to segregate cases of bipolar and unipolar depression [52,53,54]. Subsequently, machine learning assisted researchers with depression from schizophrenia and psychosis [44, 75]. These studies have found machine learning paradigms extremely useful for diagnostic studies and clinical decisions, where ever diagnostic circumstances are unclear. This appears to be a promising research direction, which is focused on enhancing the clinical utility of cases, where diagnoses are vague. Whenever diagnostic assessments are complicated, time-consuming, or costly, then machine learning could be the best option to look forward to [47].
Psychological traits of individuals could be easily identified through digital footprints with a high degree of accuracy. Behaviour on social networking platforms can reveal individual characteristics like sexual and political orientation and ethnicity. There are algorithms which can make personality predictions even better than acquaintances [55,56,57]. Studies reveal the fact that generic facial recognition algorithms can distinguish between hetero and homosexual orientation with a precision ranging from 71 to 81 percent. Further, the linguistic contents shared over social media posts could predict personality traits, gender and the linguistic features [58,59,60,61]. A variety of papers have documented the possibility of detecting mental health states from social media [60, 62] as well as physical health issues of communities such as heart disease.
Technology can facilitate continuous monitoring of psychoemotional state for individuals at high risk. The world community is looking for smart solutions to identify indices of cardiometabolic risks associated with stress reliably. Semiotics is one such project funded by European Union to develop a system for early assessment of depressive symptoms based on visual cues and facial expressions [63, 64]. The depressive disorder could even be revealed through numerous non-verbal signs [65, 66]. Sudden variance in the tonic activity of facial muscles, skin conductance, pulse rate often signifies intense emotions, which may help to identify depressive behaviour. Few of the researchers have noticed depressive disorder patterns even within electroencephalographic recordings [67]. Functional Near-Infrared Spectroscopy (fNIRS) has also been experimented with [68, 70]. The speech demonstrates a prominent nonverbal channel depicting the mental state of the speaker, which may include dominant attributes of depressive behaviour [71]. Depression being is a prominent mood disorder, predominantly affect facial appearance as well as body posture [65, 66]. Salient facial features like eyes, mouth, and eyebrows are predominant in accessing depression. Study of pupil dilation is one of the prominent areas of research. It was found that pupillary responses for positive stimuli were faster in non-depressive patients in comparison to negative stimuli [72]. Contrarily, depression patients display slow pupillary responses for positive stimuli with reduced cognitive load [73, 74, 76,77,78, 108]. Subsequently, attentional and pupil bias have also been investigated to predict depression symptoms. Saccadic eye movements in terms of latency and duration also differ for depressed and healthy participants [79, 80]. Further, facial action units were studies in terms of frequency of occurrence, mean duration and onset/ offset ratios for assessment of depression [81, 82]. Numerous studies have reported promising results on the application of action units to automatic depression assessment [7, 8, 83,84,85,86,87,88,89,90,91,92,93]. Typical facial expressions of varying intensity have been found associated with basic emotions like happy, surprise, sadness, disgust, anger, and fear. Measured intensity and frequency of these primary expressions have found to be low in case of depressed individuals [77, 83, 84, 91, 94,95,96,97]. Further, facial features like eyes and mouth were also studied along with gaze direction, averting eye contact, lazy eyelid activity, less blinking of eyes, reduced iris moment, reduced smile intensity, duration and mouth animation [98,99,100,101,102,103,104].
From the literature cited, it was observed that algorithms proposed to identify depressive individuals not only offer satisfactory results but also have better predictive power than those of friend ratings or other established procedures. Further, these algorithms provide an excellent way to collect data using machine learning algorithms. Furthermore, the published research sans recorded intensities of various expressions for depression patients. Moreover, there appears a need to associate and validate recorded emotive concentrations of depressive patients in comparison to non-depressive individuals.
3 Methods
The methodology is crucial for any experiment about the identification and treatment of Major Depressive Disorder (MDD). Attention Deficit Hyperactivity Disorder (ADHD) questionnaire [105, 106] was briefed to a group of four hundred one volunteers pursuing engineering education and between the age group of nineteen to twenty-three years. There were 254 male and 147 female volunteers. Subsequently, ADHD questionnaire was distributed amongst volunteer participants. The filled-in ADHD responses were studied by practicing clinical psychologists, to find out potential patients of depressive disorder. Amongst three hundred and eighty-seven ADHD questionnaire responses received, seventy-two respondents were identified as potential patients of depressive disorder. Out of theses anticipated depression patients, thirty-eight were called for a personal assessment/ interview as per DSM-IV [17] criteria. The assessment of thirty-eight volunteers made by clinical psychologists, confirmed eighteen cases of depressive disorder while twenty were declared not depressive. Eighteen confirmed patients were shown Amsterdam Dynamic Facial Expression Set (ADFES) as a stimulus. ADFES offers annotated faces displaying five basic emotions. Images each of angry, happy, sad, surprised, and disgusted expressions were presented successively on a computer screen for 500 ms, followed an immediate blank screen. The faces had variable intensities of each emotion from 0% full emotion to 100% full emotion, in 25% steps. Neutral facial expressions were also presented, giving a total of 120 facial presentations. Sample images from ADFES are shown in Fig. 1. Participants were instructed to press one of six labeled buttons on a response pad (the five emotions and neutral) as quickly and as accurately as possible. Their facial expressions and relevant emotions were videos graphed while they were responding to ADFES stimulus inputs.
Sample images from ADFES dataset used as the stimulus
Subsequently, the same procedure was repeated for twenty proven non-depressed individuals. Frequency of recorded facial expressions and quantitative estimates of emotions expressed against the given stimulus were used to train a computer model. Facial emotion recognition of participants was done using Convolutional Neural Network (CNN), while facial features were identified using Dlib Machine Learning library [107]. The architecture of CNN model used is (Conv 5 × 5, Maxpool 2 × 1, Dropout, Conv2 3 × 3, Maxpool2 2 × 1, Dropout, Conv3 3 × 3, maxpool3 2 × 1, Dense, Softmax). This experiment does not consider the analysis of vocal responses or any other biometric like Electrocardiography, skin conductance, pulse rate, eye gaze tracking, or Electroencephalography as they contribute towards the future scope of this experiment.
In the experiment, a Hikvision DS-2CE1AD0T-IRPF 2MP (1080P) camera was used to record the facial expressions. The resolution of the camera photo is 1920 × 1080 pixels. The resolution of camera video is 1080P, 25fps. The experiment was conducted in a well-lit room. The participations were seated at a distance of 1 meter in front of the camera. A monitor screen which will play the experiment content like basic facial expressions and emotional pictures to participations is placed in front of the subjects. The camera is beside the monitor, and the deviation angle is less than 10 degrees. At the same time, the head of participations and the collection equipment are at the same height. Facial landmarks can prove to be vital for facial expression analysis as shown in Fig. 2. It is driven by algorithms, which localize fiducial points of the face to mark and identify facial features. Constraint Local Models method, Active Shape Models, Active Appearance Model, 3D Landmark Modelling Matching, Elastic Bunch Graph Matching, and Landmark Distribution model are few prominent methods for identification of facial landmarks and emotion recognition [16, 109]. Further, facial landmark data have been analysed as time-series data. Landmark coordinates, along with displacement, acceleration, and velocity have been used as features for depression analysis. Displacement of each landmark from the mid horizontal axis signifies the motion relevant features. Action Units (AU) represent the coordinated activity of facial muscles while communicating actions and expressing emotions. AUs can be used to measure the variability of facial expressions. These AUs are fundamental to EMFACS (Emotional Facial Action Coding System) [110]. This research tries to follow an opposite approach in the identification of depressive disorder. We can anticipate behavioural attributes of depression patients concerning a video stimulus, instead of directly asking a dedicated set of questions from them. Recorded emotions, their intensities, and frequencies for depressive patients concerning ADFES stimulus were subsequently used to train yet another computer model. An algorithmic description of the proposed methodology is shown in Fig. 3. Remaining thirty-four potential depression patients, identified during the survey of three hundred and eighty-seven volunteers and were not called for personal evaluation by practicing psychologists, were called for examination using the CNN trained computer model. These thirty-four individuals were made to watch ADFES stimulus, while their facial expressions were simultaneously recorded. The volunteers need not to press any button to provide feedback, rather. Recorded videos of these volunteers were further analysed in terms of emotive responses recorded against the given stimulus. At times, when there is limited availability of expert psychologists, diagnosis of depressive disorder could become a major concern. The contemporary world is not only competitive but is also demanding. Individuals often struggle to share their mindset and opinions, hence become depressive. This situation warrants the need for a reliable, efficient, affordable, and believable system for early detection of depression symptoms. An algorithmic description of the proposed methodology is given below in Algorithm 1
Facial features identified using Dlib machine learning library
Recorded emotion intensities of a respondent suffering from depressive disorder
4 Results
This experiment was undertaken in two phases; the first phase of the experiment deals with showing ADFES stimulus to depressive and non-depressive individuals and recording of their emotive facial response. The recorded facial videos of confirmed patients of depressive disorder and non-depressive individuals, along with the results of ERT were used to train an ERT_CNN model. Facial emotions were classified using Dlib-ML library as given below:
4.1 Facial emotion recognition
One subject was made to perform emotion recognition task for 6 s 25fps on ADFES dataset. Total frames per person: 25∗6 = 150. Each frame was analysed for 6 emotions & their intensity.
-
In fi-ith-frame in the sequence
-
In Hafi-Happy emotion intensity of ith frame
-
In Safi-Sad emotion intensity of ith frame
-
Sufi-Surprise emotion intensity of ith frame
-
Anfi-Angry emotion intensity of ith frame
-
Difi-Disgust emotion intensity of ith frame
-
Nefi-Neutral emotion intensity of ith frame
-
Intensity formula = Probability of prediction of CNN
Recorded emotions of respondents as identified by DlibML library are shown in Figs. 3 and 4. Training and validation phase accuracy are loss are depicted in Figs. 5 and 6.
Recorded emotion intensities of a non-depressive respondent Origin software
CNN accuracy for training and validation
CNN loss for training and validation
Emotions and their measured intensities from recorded videos of patients suffering from a depressive disorder, and non-depressed respondents are shown in Tables 1 and 2, respectively. Emotion intensity of Overall intensity per emotion (X) is calculated as Emotion (E)∗ Intensity of emotion X (In-X) divided by 150, i.e., (X) = E(In-X)/150. Figures 7 and 8 corresponds to the graphical representation of recorded emotional intensities.
Graphical representation of observed emotions through facial analysis of depressive respondents
Graphical representation of observed emotions through facial analysis of non-depressive respondents
Owing to the required brevity of the manuscript, the emotive responses, as expressed by depressive patients and non-depressive individuals against ADFES stimulus are shown in the following tables. As stated previously, ADFES stimulus is categorized into five sets of annotated emotive intensities ranging from 0% to full emotion to 100% with an increment of 20% emotional intensity in every set. Tables 3, 4, 5, 6 and 7 corresponds to the recorded response of proven patients of depressive disorder. Tables 8, 9, 10, 11 and 12 corresponds to the recorded response of non-depressive individuals against ADFES stimulus.
Training (18 subjects –> proven depressed subjects)
Training (20 subjects proven non-depressed individuals)
An Emo_CNN classifier was trained using data collected about Tables 3–12. Based on the recorded responses, this classifier is programmed to identify any respondent as depressive or non-depressive. Subsequently, the intensity and frequency of recorded emotions were used to calculate the emotive score. Amongst seventy-two potential depressive patients identified during the pilot survey as per ADHD questionnaire, only thirty-eight were initially called for examination by qualified professionals. Remaining thirty-four individuals were subsequently called and made to repeat the process of recording their response against ADFES dataset and their facial expressions were recorded too in parallel. Their recorded responses at various intensities of emotive stimulus from ADFES dataset are shown in Tables 13, 14, 15, 16 and 17.
Testing (34 subjects)
Data collected from thirty-four individuals/ respondents were fed into ERT_CNN classifier, so as to classify respondents as potentially depressed or non-depressed. This data was used for testing the accuracy of the proposed method. To ascertain the accuracy and precision of ERT_CNN, entire of the respondents were subsequently examined by practicing psychologists, as shown in Table 19. It was noticed that, amongst fifteen respondents, which were classified as depressed, ten were actually found suffering from a depressive disorder. Twelve of the remaining respondents were correctly identified as non-depressed, whereas seven were wrongly classified as depressed. Table 18 corresponds to the observed emotional intensities of unconfirmed patients of depressive disorder. Graphical representation of observed intensities is shown in Fig. 9. The proposed method attained a precision of 64 in the identification of patients suffering from a depressive disorder as given in the confusion matrix shown in Table 19. Though the observed results are not outstanding, yet they are promising. Being at the experimental stage, the precision attained by the proposed method seems satisfactory. Imbuing facial emotions with time-proven techniques like ADHD questionnaire for diagnosing depressive disorders is not the only novel but also has ample scope of improvement too.
Graphical representation of observed emotions through facial analysis of potentially depressive respondents
Confusion matrix of ERT_CNN
Total test subjects: 34
Depressed: 15
Non depressed: 19
Precision = (12/15 + 14/19)/2 = (0.8 + 0.68)/2 = 0.74
5 Conclusion
his study concludes to the fact that facial expressions rendered by a patient suffering from the depressive disorder are different from that of a normal person against any given psychological stimulus. Further, it was also concluded that facial expressions rendered by a respondent against any annotated quality stimulus like ADFES dataset could provide results comparable to that of ADHD questionnaire. Facial analysis of respondents promulgated the fact that sad and neutral were the most prominent emotions displayed by depressive patents. On the contrary, non-depressive respondents enjoyed while responding to ADFES stimulus as their facial analysis identified happy as the most prominent emotion, displayed by any respondent, followed by neutral and surprised. It was also noticed that at depressive patients, wrongly identified lower intensity emotional stimulus from ADFES dataset. Their responses became somewhat better with high-intensity emotional stimulus from ADFES dataset, but the overall emotional intensity displayed by any depressive patient was far lower than any non-depressive respondent. This experiment is oriented towards diagnosing the depressive disorder based upon emotive display of respondents; at this stage, it cannot differentiate between depression and anxiety. Further, emotions displayed through facial movements only may not suffice to augment the precision of the proposed methodology. The future scope of this study lies with the deployment of other technologies like Electroencephalography, Galvanic skin response and pulse sensors, etc. along with an understanding of emotions through facial analysis to diagnose a person with depressive disorder. The proposed method could be used for self-diagnosis of depressive disorder within masses, or it could be of great help for a practicing psychologist to better understand the emotive state of any prospective patient seeking help.
References
Adorni R, Gatti A, Brugnera A, Sakatani K, Compare A (2016) Frontiers in psychology. Front Psychol 7. https://psycnet.apa.org/record/2016-16013-001 (August 7, 2019)
Alghowinem S et al (2013) Eye movement analysis for depression detection. In: 2013 IEEE International Conference on Image Processing. http://ieeexplore.ieee.org/document/6738869/ (August 8, 2019). IEEE, pp 4220–4224
Alghowinem S, Goecke R, Cohn JF, Wagner M, Parker G, Breakspear M (2019) Cross-cultural detection of depression from nonverbal behaviour. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG). http://ieeexplore.ieee.org/document/7163113/ (August 8, 2019). IEEE, pp 1–8
American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders. American Psychiatric Association. https://psychiatryonline.org/doi/book/10.1176/appi.books.9780890425596 (August 5, 2019)
Amsterdam Interdisciplinary Centre for Emotion (AICE) (2019) https://aice.uva.nl/research-tools/adfes-stimulus-set/adfes-stimulus-set.htm
Arbabshirani MR, Plis S, Sui J, Calhoun VD (2017) Single subject prediction of brain disorders in neuroimaging: promises and pitfalls. NeuroImage 145(Pt B):137–165. http://www.ncbi.nlm.nih.gov/pubmed/27012503 (August 6, 2019)
Automatic audiovisual behavior descriptors for psychological disorder analysis. Image Vis Comput 32(10):648–658. https://www.sciencedirect.com/science/article/pii/S0262885614001000?via%3Dihub (August 8, 2019)
Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender. J Multimodal User Interfaces 9(1):17–29. http://springerlink.bibliotecabuap.elogim.com/10.1007/s12193-014-0161-4 (August 7, 2019)
Bennett K, Bennett AJ, Griffiths KM (2010) Security considerations for E-Mental health interventions. J Med Internet Res 12 (5):e61. http://www.ncbi.nlm.nih.gov/pubmed/21169173 (August 5, 2019)
Beyond group differences. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge - AVEC ’13. ACM Press, New York, 1–2. http://dl.acm.org/citation.cfm?doid=2512530.2512537 (August 7, 2019)
Bittner A et al (2007) What do childhood anxiety disorders predict? J Child Psychol Psychiatry 48(12):1174–1183. http://www.ncbi.nlm.nih.gov/pubmed/18093022 (August 5, 2019)
Bohannon J (2015) The synthetic therapist. Science 349 (6245):250–251. http://www.ncbi.nlm.nih.gov/pubmed/26185240 (August 6, 2019)
Bufferd SJ, Dougherty LR, Carlson GA, Klein DN (2011) Parent-reported mental health in preschoolers: findings using a diagnostic interview. Compr Psychiatry 52(4):359–369. https://www.sciencedirect.com/science/article/abs/pii/S0010440X10001446 (August 2, 2019)
Carcione A et al (2008) An intensive case analysis of client metacognition in a good-outcome psychotherapy: Lisa’s case. Psychother Res 18(6):667–676. http://www.ncbi.nlm.nih.gov/pubmed/18815952 (August 5, 2019)
Chandra GS et al (2017) Detecting depression and mental illness on social media: an integrative review. Curr Opin Behav Sci 18:43–49. https://doi.org/10.1016/j.cobeha.2017.07.005 (August 6, 2019)
Chandrashekar P (2018) Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. mHealth 4. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5897664/ (August 5, 2019)
Chiarugi F et al (2014) Facial signs and psycho-physical status estimation for well-being assessment. In: Proceedings of the international conference on health informatics, SCITEPRESS—science and and technology publications. http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0004934405550562 (August 7, 2019), pp 555–562
Cohn JF (2010) Social signal processing in depression. In: Proceedings of the 2nd international workshop on social signal processing—SSPW ’10. http://portal.acm.org/citation.cfm?doid=1878116.1878118 (August 7, 2019). ACM Press, New York
Cohn JF et al (2009) Detecting depression from facial actions and vocal prosody. In: 2009 3rd International conference on affective computing and intelligent interaction and workshops. http://ieeexplore.ieee.org/document/5349358/ (August 7, 2019). IEEE, pp 1–7
Cortes C, Vapnik V (1995) Support vector networks. Mach Learn 20(3):273–297
Coryell W et al (1993) The enduring psychosocial consequences of mania and depression. Am J Psychiatry 150(5):720–727. http://www.ncbi.nlm.nih.gov/pubmed/8480816 (August 5, 2019)
Craske MG, Stein MB (2016) Anxiety. Lancet (London England) 388(10063):3048–3059. http://www.ncbi.nlm.nih.gov/pubmed/27349358 (August 5, 2019)
Csernansky JG et al (2004) Abnormalities of thalamic volume and shape in Schizophrenia. Am J Psychiatry 161(5):896–902. http://www.ncbi.nlm.nih.gov/pubmed/15121656 (August 6, 2019)
Cummins N et al (2015) A review of depression and suicide risk assessment using speech analysis. Speech Commun 71(C):10–49. https://linkinghub.elsevier.com/retrieve/pii/S0167639315000369 (August 8, 2019)
Davatzikos C et al (2008) Detection of prodromal Alzheimer’s disease via pattern classification of magnetic resonance imaging. Neurobiol Aging 29 (4):514–523. http://www.ncbi.nlm.nih.gov/pubmed/17174012 (August 6, 2019)
Egger HL, Angold A (2006) Common emotional and behavioral disorders in preschool children: presentation, nosology, and epidemiology. J Child Psychol Psychiatry 47(3–4):313–337. http://doi.wiley.com/10.1111/j.1469-7610.2006.01618.x (August 2, 2019)
Eichstaedt JC et al (2015) Psychological language on twitter predicts county-level heart disease mortality. Psychol Sci 26(2):159–169. http://www.ncbi.nlm.nih.gov/pubmed/25605707 (August 6, 2019)
Ekman P, Friesen WV (1976) Pictures of facial affect. Consulting Psychologists Press
Ekman P, Friesen WV, Hager JC (2002) Facial action coding system. Manual and investigator’s guide UT. Salt Lake City, Research Nexus
Ellgring H (2007) Non-verbal communication in depression. Cambridge University Press, Cambridge. https://www.bookdepository.com/European-Monographs-Social-Psychology-Non-verbal-Communication-Depression-Heiner-Ellgring/9780521047562 (August 7, 2019)
Ellgring H (2008) Non-verbal communication in depression. Cambridge University Press, Cambridge
Epstein J, Klinkenberg WD (2001) From Eliza to Internet: a brief history of computerized assessment. Comput Hum Behav 17(3):295–314. https://www.sciencedirect.com/science/article/pii/S0747563201000048 (August 5, 2019)
Fajutrao L, Locklear J, Priaulx J, Heyes A (2009) A systematic review of the evidence of the burden of bipolar disorder in Europe. Clinical practice and epidemiology in mental health: CP & EMH 5:3. http://www.ncbi.nlm.nih.gov/pubmed/19166608 (August 6, 2019)
Firth J et al (2017) The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 16(3):287–298. http://www.ncbi.nlm.nih.gov/pubmed/28941113 (August 5, 2019)
Fiske ST, Taylor SE (1991) Social cognition, 2nd edn. McGraw-Hill. https://psycnet.apa.org/record/1991-97723-000 (August 5, 2019)
Fu CHY et al (2008) Pattern classification of sad facial processing: toward the development of neurobiological markers in depression. Biol Psychiatry 63 (7):656–662. http://www.ncbi.nlm.nih.gov/pubmed/17949689 (August 6, 2019)
Ghosh S, Chatterjee M, Morency L -P (2014) A multimodal context-based approach for distress assessment. In: Proceedings of the 16th international conference on multimodal interaction—ICMI ’14. http://dl.acm.org/citation.cfm?doid=2663204.2663274 (August 7, 2019). ACM Press, New York, pp 240–246
Girard JM et al (2013) Social risk and depression: evidence from manual and automatic facial expression analysis. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). http://ieeexplore.ieee.org/document/6553748/ (August 7, 2019). IEEE, pp 1–8
Gould MS et al (1998) Psychopathology associated with suicidal ideation and attempts among children and adolescents. J Am Acad Child Adolesc Psychiatry 37(9):915–923. http://www.ncbi.nlm.nih.gov/pubmed/9735611 (August 5, 2019)
Gratch J et al (2014) The distress analysis interview corpus of human and computer interviews. In: European languages resources association (ELRA). https://aclweb.org/anthology/papers/L/L14/L14-1421/ (August 8, 2019), pp 3123–3128
Griffiths KM, Christensen H (2000) Quality of web based information on treatment of depression: cross sectional survey. BMJ 321(7275):1511–1515. http://www.ncbi.nlm.nih.gov/pubmed/11118181 (August 5, 2019)
Gupta R et al (2014) Multimodal prediction of affective dimensions and depression in human-computer interactions. In: Proceedings of the 4th international workshop on audio/visual emotion challenge—AVEC ’14. http://dl.acm.org/citation.cfm?doid=2661806.2661810 (August 8, 2019). ACM Press, New York, pp 33–40
Hess U, Blairy S, Kleck RE (2000) The influence of facial emotion displays, gender, and ethnicity on judgments of dominance and affiliation. J Nonverbal Behav 24(4):265–283. http://springerlink.bibliotecabuap.elogim.com/10.1023/A:1006623213355 (August 5, 2019)
Hess U, Adams R, Kleck R (2005) Who may frown and who should smile? Dominance, affiliation, and the display of happiness and anger. Cognit Emot 19(4):515–536. http://www.tandfonline.com/doi/abs/10.1080/02699930441000364 (August 5, 2019)
Hirschfeld RM et al (2000) Social functioning in depression: a review. J Clin Psychiatry 61(4):268–275. http://www.ncbi.nlm.nih.gov/pubmed/10830147 (August 5, 2019)
Hirschfeld RMA, Lewis L, Vornik LA (2003) Perceptions and impact of bipolar disorder: how far have we really come? Results of the national depressive and manic-depressive association 2000 survey of individuals with bipolar disorder. J Clin Psychiatry 64(2):161–174. http://www.ncbi.nlm.nih.gov/pubmed/12633125 (August 6, 2019)
Hosseinifard B, Moradi MH, Rostami R (2013) Classifying depression patients and normal subjects using machine learning techniques and nonlinear features from EEG signal. Comput Methods Progr Biomed 109(3):339–345. http://www.ncbi.nlm.nih.gov/pubmed/23122719 (August 7, 2019)
Jans M, Soffer P, Jouck T (2019) Building a valuable event log for process mining: an experimental exploration of a guided process. Enterprise Inf Syst 1–30. https://www.tandfonline.com/doi/full/10.1080/17517575.2019.1587788
Jones NP, Siegle GJ, Mandell D (2015) Motivational and emotional influences on cognitive control in depression: a pupillometry study. Cognit Affect Behav Neurosci 15(2):263–275. http://www.ncbi.nlm.nih.gov/pubmed/25280561 (August 7, 2019)
Judd LL et al (2000) Psychosocial disability during the long-term course of unipolar major depressive disorder. Arch Gen Psychiatry 57(4):375. http://www.ncbi.nlm.nih.gov/pubmed/10768699 (August 5, 2019)
Kambeitz J et al (2017) Detecting neuroimaging biomarkers for depression: a meta-analysis of multivariate pattern recognition studies. Biol Psychiatry 82(5):330–338. http://www.ncbi.nlm.nih.gov/pubmed/28110823 (August 6, 2019)
Khazaal Y et al (2008) Quality of web-based information on social phobia: a cross-sectional study. Depress Anxiety 25(5):461–465. http://www.ncbi.nlm.nih.gov/pubmed/17960640 (August 5, 2019)
King DE (2009) Dlib-Ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758. http://jmlr.csail.mit.edu/papers/v10/king09a.html
Kloppel S et al (2008) Accuracy of dementia diagnosis—a direct comparison between radiologists and a computerized method. Brain 131(11):2969–2974. https://academic.oup.com/brain/article-lookup/doi/10.1093/brain/awn239 (August 6, 2019)
Knutson B (1996) Facial expressions of emotion influence interpersonal trait inferences. J Nonverbal Behav 20(3):165–182. http://springerlink.bibliotecabuap.elogim.com/10.1007/BF02281954 (August 5, 2019)
Kosinski M, Stillwell D, Graepel T (2013) Private traits and attributes are predictable from digital records of human behavior. Proc Natl Acad Sci 110(15):5802–5805. http://www.pnas.org/cgi/doi/10.1073/pnas.1218772110 (August 6, 2019)
Koutsouleris N et al (2015) Individualized differential diagnosis of schizophrenia and mood disorders using neuroanatomical biomarkers. Brain 138 (7):2059–2073. http://www.ncbi.nlm.nih.gov/pubmed/25935725 (August 6, 2019)
Kudinova AY et al (2016) Pupillary reactivity to negative stimuli prospectively predicts recurrence of major depressive disorder in women. https://binghamton.edu/psychology/labs/mood/pdfs/2016_kudinova_pupilpredicts_mddrecurrence_inwomen.pdf (August 7, 2019)
Ladegaard N, Larsen ER, Videbech P, Lysaker PH (2014) Higher-order social cognition in first-episode major depression. Psychiatry Res 216 (1):37–43. http://www.ncbi.nlm.nih.gov/pubmed/24524945 (August 5, 2019)
Ladegaard N, Lysaker PH, Larsen ER, Videbech P (2014) A comparison of capacities for social cognition and metacognition in first episode and prolonged depression. Psychiatry Res 220(3):883–889. http://www.ncbi.nlm.nih.gov/pubmed/25453639 (August 5, 2019)
Lavagnino L et al (2015) Identifying neuroanatomical signatures of anorexia nervosa: a multivariate machine learning approach. Psychol Med 45 (13):2805–2812. https://www.cambridge.org/core/product/identifier/S0033291715000768/type/journal_article (August 6, 2019)
Li M et al (2016) Alleviated negative rather than positive attentional bias in patients with depression in remission: an eye-tracking study. J Int Med Res 44 (5):1072–1086. http://journals.sagepub.com/doi/10.1177/0300060516662134 (August 7, 2019)
Lindefors N, Andersson G (2016) Guided internet-based treatments in psychiatry
Luby JL et al (2009) The clinical significance of preschool depression: impairment in functioning and clinical markers of the disorder. J Affect Disord 112(1–3):111–119. https://www.sciencedirect.com/science/article/abs/pii/S0165032708001481 (August 5, 2019)
Lucas GM et al (2015) Towards an affective interface for assessment of psychological distress. In: 2015 International conference on affective computing and intelligent interaction (ACII). http://ieeexplore.ieee.org/document/7344622/ (August 8, 2019). IEEE, pp 539–545
Lueken U et al (2015) Separating depressive comorbidity from panic disorder: a combined functional magnetic resonance imaging and machine learning approach. J Affect Disord 184:182–192. http://www.ncbi.nlm.nih.gov/pubmed/26093832 (August 6, 2019)
Marquand AF, Rezek I, Buitelaar J, Beckmann CF (2016) Understanding heterogeneity in clinical cohorts using normative models: beyond case-control studies. Biol Psychiatry 80(7):552–561. http://www.ncbi.nlm.nih.gov/pubmed/26927419 (August 6, 2019)
Matsumoto D, Ekman P (2008) Facial expression analysis. Scholarpedia 3(5):4237. http://www.scholarpedia.org/article/Facial_expression_analysis (August 5, 2019)
McIntyre GJ (2010) The computer analysis of facial expressions: on the example of depression and anxiety. http://users.cecs.anu.edu.au/~gmcintyr/thesis.pdf (August 7, 2019)
McIntyre G, Goecke R, Breakspear M, Parker G (2011) Facial response to video content in depression 1–2. https://researchprofiles.canberra.edu.au/en/publications/facial-response-to-video-content-in-depression(August 7, 2019)
Mehrabian A, Russell JA (1974) An approach to environmental psychology. PsycNET. The MIT Press, Cambridge. https://psycnet.apa.org/record/1974-22049-000 (August 5, 2019)
Mohr DC, Zhang M, Schueller SM (2017) Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annu Rev Clin Psychol 13(1):23–47. http://www.ncbi.nlm.nih.gov/pubmed/28375728 (August 5, 2019)
Morency L -P et al (2015) SimSensei demonstration: a perceptive virtual human interviewer for healthcare applications. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence, pp 4307–4308
Nielssen O et al (2015) Procedures for risk management and a review of crisis referrals from the MindSpot Clinic, a national service for the remote assessment and treatment of anxiety and depression. BMC Psychiatry 15(1):304. http://www.ncbi.nlm.nih.gov/pubmed/26626712 (August 5, 2019)
Nonverbal social withdrawal in depression: evidence from manual and automatic analyses. Image Vis Comput 32(10):641–647. https://linkinghub.elsevier.com/retrieve/pii/S0262885613001790 (August 7, 2019)
Orrù G et al (2012) Using support vector machine to identify imaging biomarkers of neurological and psychiatric disease: a critical review. Neurosci Biobehav Rev 36(4):1140–1152. http://www.ncbi.nlm.nih.gov/pubmed/22305994 (August 6, 2019)
Pampouchidou A et al (2017) Automatic assessment of depression based on visual cues: a systematic review. IEEE Transactions on Affective Computing 1–1. http://ieeexplore.ieee.org/document/8052569/ (August 2, 2019)
Park G et al (2015) Automatic personality assessment through social media language. J Person Social Psychol 108(6):934–952. http://www.ncbi.nlm.nih.gov/pubmed/25365036 (August 6, 2019)
Pediaditis M et al (2015) Extraction of facial features as indicators of stress and anxiety. In: 2015 37th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC). http://ieeexplore.ieee.org/document/7319199/ (August 7, 2019). IEEE, pp 3711–3714
Poria S, Mondal A, Mukhopadhyay P (2015) Evaluation of the intricacies of emotional facial expression of psychiatric patients using computational models. In: Understanding facial expressions in communication. http://springerlink.bibliotecabuap.elogim.com/10.1007/978-81-322-1934-7_10 (August 7, 2019). Springer India, New Delhi, pp 199–226
Redlich R et al (2014) Brain morphometric biomarkers distinguishing unipolar and bipolar depression. JAMA Psychiatry 71(11):1222. http://www.ncbi.nlm.nih.gov/pubmed/25188810 (August 6, 2019)
Russell JA, Bullock M (1985) Multidimensional scaling of emotional facial expressions: similarity from preschoolers to adults. J Personal Social Psychol 48(5):1290–1298. http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-3514.48.5.1290 (August 5, 2019)
Saragih JM, Lucey S, Cohn JF (2009) Face alignment through subspace constrained mean-shifts. In: 2009 IEEE 12th international conference on computer vision. http://ieeexplore.ieee.org/document/5459377/ (August 8, 2019). IEEE, pp 1034–1041
Scherer S et al (2013) Automatic behavior descriptors for psychological disorder analysis. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). http://ieeexplore.ieee.org/document/6553789/ (August 8, 2019). IEEE, pp 1–8
Scherer S, Stratou G, Morency L-P (2013) Audiovisual behavior descriptors for depression assessment. In: Proceedings of the 15th ACM on international conference on multimodal interaction—ICMI ’13. http://dl.acm.org/citation.cfm?doid=2522848.2522886 (August 8, 2019). ACM Press, New York, pp 135–140
Siegle GJ et al (2011) Remission prognosis for cognitive therapy for recurrent depression using the pupil: utility and neural correlates. Biol Psychiatry 69(8):726–733. http://www.ncbi.nlm.nih.gov/pubmed/21447417 (August 7, 2019)
Silk JS et al (2007) Pupillary reactivity to emotional information in child and adolescent depression: links to clinical and ecological measures. Am J Psychiatry 164(12):1873–1880. http://www.ncbi.nlm.nih.gov/pubmed/18056243 (August 7, 2019)
Stratou G, Scherer S, Gratch J, Morency L-P (2013) Automatic nonverbal behavior indicators of depression and PTSD: exploring gender differences. In: 2013 Humaine association conference on affective computing and intelligent interaction. http://ieeexplore.ieee.org/document/6681422/ (August 7, 2019). IEEE, pp 147–152
Suto T et al (2004) Multichannel near-infrared spectroscopy in depression and Schizophrenia: cognitive brain activation study. Biol Psychiatry 55 (5):501–511. http://www.ncbi.nlm.nih.gov/pubmed/15023578 (August 7, 2019)
Valstar M et al (2016) AVEC 2016—depression, mood, and emotion recognition workshop and challenge. 1605.01600 (August 7, 2019)
van der Schalk J, Hawk ST, Fischer AH, Doosje B (2011) Moving faces, looking places: validation of the amsterdam dynamic facial expression set (ADFES). Emotion 11(4):907–920. http://doi.apa.org/getdoi.cfm?doi=10.1037/a0023853 (August 8, 2019)
Visser RM et al (2016) First steps in using multi-voxel pattern analysis to disentangle neural processes underlying generalization of spider fear. Front Hum Neurosci 10:222. http://www.ncbi.nlm.nih.gov/pubmed/27303278 (August 6, 2019)
Vlaescu G et al (2016) Features and functionality of the Iterapi platform for internet-based psychological treatment. Internet Interv 6:107–114. http://www.ncbi.nlm.nih.gov/pubmed/30135819 (August 5, 2019)
Wang Y, Kosinski M (2018) Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. J Personal Social Psychol 114(2):246–257. http://doi.apa.org/getdoi.cfm?doi=10.1037/pspa0000098 (August 6, 2019)
Wang J et al (2014) Pupillometry in Chinese female patients with depression: a pilot study. Int J Environ Res Public Health 11 (2):2236–2243. http://www.mdpi.com/1660-4601/11/2/2236 (August 7, 2019)
Wang Q, Yang H, Yu Y (2018) Facial expression video analysis for depression detection in Chinese patients. J Vis Commun Image Represent 57:228–233. https://doi.org/10.1016/j.jvcir.2018.11.003
Watson D, Clark LA, Carey G (1988) Positive and negative affectivity and their relation to anxiety and depressive disorders. J Abnorm Psychol 97 (3):346–353. http://www.ncbi.nlm.nih.gov/pubmed/3192830 (August 5, 2019)
Waxer PH (1974) Therapist training in nonverbal communication I: nonverbal cues for depression. J Clin Psychol 30(2):215–218. http://doi.wiley.com/10.1002/1097-4679%28197404%2930%3A2%3C215%3A%3AAID-JCLP2270300229%3E3.0.CO%3B2-Q (August 7, 2019)
Whalen DJ, Sylvester CM, Luby JL (2017) Depression and anxiety in preschoolers: a review of the past 7 years. Child Adolesc Psychiatr Clin North America 26(3):503–522. http://www.ncbi.nlm.nih.gov/pubmed/28577606 (August 5, 2019)
Whelan R, Garavan H (2014) When optimism hurts: inflated predictions in psychiatric neuroimaging. Biol Psychiatry 75(9):746–748. http://www.ncbi.nlm.nih.gov/pubmed/23778288 (August 6, 2019)
Williamson JR et al (2014) Vocal and facial biomarkers of depression based on motor incoordination and timing. In: Proceedings of the 4th international workshop on audio/visual emotion challenge—AVEC ’14. http://dl.acm.org/citation.cfm?doid=2661806.2661809 (August 7, 2019). ACM Press, New York, pp 65–72
Winograd-Gurvich C et al (2006) Ocular motor differences between melancholic and non-melancholic depression. J Affect Disord 93(1–3):193–203. https://www.sciencedirect.com/science/article/abs/pii/S0165032706001431 (August 7, 2019)
Winograd-Gurvich C et al (2006) Self-paced and reprogrammed saccades: differences between melancholic and non-melancholic depression. Neurosci Res 56(3):253–260. http://www.ncbi.nlm.nih.gov/pubmed/16914221 (August 7, 2019)
Wittchen H-U, Sonntag H (2000) Nicotine consumption in mental disorders: a clinical epidemiological perspective. Eur Neuropsychopharmacol 10:119. https://linkinghub.elsevier.com/retrieve/pii/S0924977X00800140 (August 5, 2019)
World Health Organization (2017) Depression and other common mental disorders. Institutes Health of National (1):1–22
Yaden DB et al (2018) The language of religious affiliation. Social Psychol Personal Sci 9 (4):444–452. http://journals.sagepub.com/doi/10.1177/1948550617711228 (August 6, 2019)
Yang T-H, Wu C-H, Huang K -Y, Su M-H (2017) Coupled HMM-based multimodal fusion for mood disorder detection through elicited audio–visual signals. J Ambient Intell Humaniz Comput 8(6):895–906. http://springerlink.bibliotecabuap.elogim.com/10.1007/s12652-016-0395-y (August 7, 2019)
Women are warmer but no less assertive than men: gender and language on Facebook ed. Christopher M. Danforth. PLOS ONE 11(5): e0155885. http://dx.plos.org/10.1371/journal.pone.0155885 (August 6, 2019)
Wu Y, Kosinski M, Stillwell D (2015) Computer-based personality judgments are more accurate than those made by humans. Proc Natl Acad Sci USA 112(4):1036–1040. http://www.ncbi.nlm.nih.gov/pubmed/25583507 (August 6, 2019)
Yu Z et al (2013) Multimodal prediction of psychological disorders: learning verbal and nonverbal commonalities in adjacency pairs. https://www.semanticscholar.org/paper/Multimodal-Prediction-of-Psychological-Disorders%3A-Yu-Scherer/4a797110c9fbf8a05f0b5747402bc85d583a864e (August 8, 2019)
Zhou D, Luo J, Silenzio V, Zhou Y, Hu J, Currier G, Kautz H (2015) Tackling mental health by integrating unobtrusive multimodal sensing. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence. https://dl.acm.org/citation.cfm?id=2887201 (August 8, 2019), p 1034
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Singh, J., Goyal, G. Decoding depressive disorder using computer vision. Multimed Tools Appl 80, 8189–8212 (2021). https://doi.org/10.1007/s11042-020-10128-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10128-9