Skip to main content
SpringerLink
Log in

Evaluation of Sequential and Temporally Embedded Deep Learning Models for Health Outcome Prediction

  • Chapter
  • First Online:
Deep Learning Applications, Volume 4

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1434))

  • 724 Accesses

Abstract

Deep learning sequential models are increasingly being used to predict patients’ health outcomes by analyzing their medical histories. In this paper, we investigate the design decisions and challenges of using deep learning sequential models for predictive health modeling. Our results show that the most successful deep learning health models to date, called transformers, lack a mechanism to analyze the temporal characteristics of health records. To address this gap, we propose and evaluate a new model called DTTHRE: Decoder Transformer for Temporally Embedded Health Records Encoding. DTTHRE analyzes patients’ medical histories, including the elapsed time between visits. We also evaluate the performance of DTTHRE on a real-world medical dataset for two health outcomes: (1) diagnostic and (2) readmission prediction. DTTHRE successfully predicted patients’ final diagnosis (78.54 ± 0.22%) and readmission risk (99.91 ± 0.02%) with improved performance compared to existing deep learning sequential models in the literature. DTTHRE predicts the health outcome for each medical visit, which increases the training examples available from limited medical datasets with no additional training time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bellos, C., Papadopoulos, A., Rosso, R., Fotiadis, D.I.: Heterogeneous data fusion and intelligent techniques embedded in a mobile application for real-time chronic disease management. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS-11) (2011). https://doi.org/10.1109/IEMBS.2011.6092047

  2. Boursalie, O., Samavi, R., Doyle, T.E.: Machine learning and mobile health monitoring platforms: a case study on research and implementation challenges. J. Healthc. Inf. Res. 2(1) (2018). https://doi.org/10.1007/s41666-018-0021-1

  3. Boursalie, O., Samavi, R., Doyle, T.E.: Decoder transformer for temporally-embedded health outcome predictions. In: IEEE The 20th International Conference on Machine Learning and Applications (ICMLA-21) (2021). https://doi.org/10.1109/ICMLA52953.2021.00235

  4. Boursalie, O., Samavi, R., Doyle, T.E.: Evaluation metrics for deep learning imputation models. In: AAAI The 5th International Workshop on Health Intelligence (W3PHIAI-21) (2021). https://doi.org/10.1007/978-3-030-93080-6_22

  5. Boursalie, O., Samavi, R., Doyle, T.E., Koff, D.A.: Deep learning model for cancer risk from low dose medical imaging radiation. In: European Congress of Radiology (ECR) (2020). https://doi.org/10.26044/esi2020/ESI-10315

  6. Boursalie, O., Samavi, R., Doyle, T.E., Koff, D.A.: Using medical imaging effective dose in deep learning models: estimation & evaluation. IEEE Trans. Radiat. Plasma. Med. Sci. (2020). https://doi.org/10.1109/TRPMS.2020.3029038

    Article  Google Scholar 

  7. Canadian Institute for Health Information (CIHI): Canadian Coding Standards for Version 2010 ICD-10-CA and CCI. Technical Report (2010)

    Google Scholar 

  8. Canadian Nuclear Safety Commission (CNSC): In: Radiation Doses (2020). https://www.cnsc-ccsn.gc.ca/eng/resources/research/index.cfm

  9. Choi, E., Bahadori, M.T., Schuetz, A., Stewart, W.F., Sun, J.: Doctor AI: predicting clinical events via recurrent neural networks. In: Machine Learning for Healthcare Conference (MLHC-16). PMLR (2016)

    Google Scholar 

  10. Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., Stewart, W.: Retain: an interpretable predictive model for healthcare using reverse time attention mechanism. In: Advances in Neural Information Processing Systems (NeurIPS-16), vol. 29 (2016)

    Google Scholar 

  11. Choi, E., Xiao, C., Stewart, W., Sun, J.: MiME: multilevel medical embedding of EHR for predictive healthcare. In: 32nd Conference on Neural Information Processing Systems (NeurIPS-18) (2018). https://papers.nips.cc/paper/2018/file/934b535800b1cba8f96a5d72f72f1611-Paper.pdf

  12. Comito, C., Forestiero, A., Papuzzo, G.: Exploiting social media to enhance clinical decision support. In: International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT-19) (2019). https://doi.org/10.1145/3358695.3360899

  13. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics (NAACL) (2019). https://aclanthology.org/N19-1423.pdf

  14. Dietterich, T.G.: Machine learning for sequential data: a review. In: Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR) (2002). https://doi.org/10.1007/3-540-70659-3_2

  15. Esfandiari, N., Babavalian, M.R., Moghadam, A.M.E., Tabar, V.K.: Knowledge discovery in medicine: current issue and future trend. Expert Syst. Appl. 41(9) (2014). https://doi.org/10.1016/j.eswa.2014.01.011

  16. Fawaz, H.I., Forestier, G., Weber, J., Idoumghar, L., Muller, P.A.: Data augmentation using synthetic data for time series classification with deep residual networks. In: ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data (AALTD-18) (2018). https://arxiv.org/pdf/1808.02455.pdf

  17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8) (1997). https://doi.org/10.1162/neco.1997.9.8.1735

  18. Huang, C., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A.M., Hoffman, M.D., Dinculescu, M., Eck, D.: Music transformer. In: International Conference on Learning Representations (ICLR-19) (2019). https://openreview.net/pdf?id=rJe4ShAcF7

  19. Huang, M., Zolnoori, M., Shah, N.D., Yao, L.: Temporal sequence alignment in EHR for patient representation. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM-18) (2018). https://doi.org/10.1109/BIBM.2018.8621428

  20. International Commission on Radiology Protection (ICRP): The 2007 Recommendations of the ICRP. ICRP Publication 103, vol. 37, no. (2–4) (2007)

    Google Scholar 

  21. Johnson, A.E., Pollard, T.J., Shen, L., Li-Wei, H.L., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L.A., Mark, R.G.: MIMIC-III, a freely accessible critical care database. Sci. Data 3(1) (2016). https://doi.org/10.1038/sdata.2016.35

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR-15) (2015). https://arxiv.org/abs/1412.6980

  23. Li, Y., Rao, S., Solares, J.R.A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., Salimi-Khorshidi, G.: BEHRT: transformer for EHR. Nat. Sci. Rep. 10(1) (2020). https://doi.org/10.1038/s41598-020-62922-y

  24. Lin, E.C.: Radiation risk from medical imaging. Mayo Clin. Proc. 85(12) (2010). https://doi.org/10.4065/mcp.2010.0260

  25. Liu, M., Jiang, M., Kawai, V.K., Stein, C.M., Roden, D.M., Denny, J.C., Xu, H.: Modeling drug exposure data in electronic medical records: an application to Warfarin. In: American Medical Informatics Association Symposium (2011). https://europepmc.org/articles/PMC3243123

  26. Miotto, R., Li, L., Kidd, B.A., Dudley, J.T.: Deep patient: an unsupervised representation to predict the future of patients from the EHR. Nat. Sci. Rep. 6(1) (2016). https://doi.org/10.1038/srep26094

  27. Mirtchouk, M., Srikishan, B., Kleinberg, S.: Hierarchical information criterion for variable abstraction. In: Machine Learning for Healthcare (MLHC-21) (2021). http://www.healthailab.org/papers/21MLHC.pdf

  28. National Research Council: Health Risks from Exposure to Low Levels of Ionizing Radiation: BEIR VII Phase 2, vol. 7. National Academies Press (2006)

    Google Scholar 

  29. Nguyen, P., Tran, T., Wickramasinghe, N., Venkatesh, S.: Deepr: a convolutional net for medical records. IEEE J. Biomed. Health Inf. 21(1), 22–30 (2016)

    Article  Google Scholar 

  30. Oore, S., Simon, I., Dieleman, S., Eck, D., Simonyan, K.: This time with feeling: learning expressive musical performance. Neural. Comput. Appl. 32(4) (2020). https://doi.org/10.1007/s00521-018-3758-9

  31. Pham, T., Tran, T., Phung, D., Venkatesh, S.: DeepCare: a deep dynamic memory model for predictive medicine. In: Advances in Knowledge Discovery and Data Mining (AKDD-16) (2016). https://doi.org/10.1007/978-3-319-31750-2_3

  32. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners. Technical Report (2019)

    Google Scholar 

  33. Rasmy, L., Xiang, Y., Xie, Z., Tao, C., Zhi, D.: Med-BERT: pretrained contextualized embeddings on large-scale structured EHR for disease prediction. npj Digit. Med. 4(1) (2021). https://doi.org/10.1038/s41746-021-00455-y

  34. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Internal Representations by Error Propagation. Technical Report, University of California San Diego (1985). https://apps.dtic.mil/sti/citations/ADA164453

  35. Schäfer, P.: Scalable time series classification. Data Min. Knowl. Discov. 30(5) (2016). https://doi.org/10.1007/s10618-015-0441-y

  36. Shickel, B., Tighe, P.J., Bihorac, A., Rashidi, P.: Deep EHR: a survey of recent advances in deep learning techniques for EHR analysis. IEEE J. Biomed. Health Inf. 22(5) (2018). https://doi.org/10.1109/JBHI.2017.2767063

  37. Teo, K., Yong, C.W., Chuah, J.H., Hum, Y.C., Tee, Y.K., Xia, K., Lai, K.W.: Current trends in readmission prediction: an overview of approaches. Arab. J. Sci. Eng. (2021). https://doi.org/10.1007/s13369-021-06040-5

    Article  Google Scholar 

  38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Conference on Neural Information Processing Systems (NeurIPS-17) (2017)

    Google Scholar 

  39. Vig, J.: A multiscale visualization of attention in the transformer model. In: 57th Annual Meeting of the Association for Computational Linguistics (ACL-19) (2019). https://aclanthology.org/P19-3007.pdf

  40. Vitolo, C., Scutari, M., Ghalaieny, M., Tucker, A., Russell, A.: Modeling air pollution, climate, and health data using Bayesian networks: a case study of the english regions. Earth Space Sci. 5(4) (2018). https://doi.org/10.1002/2017EA000326

  41. Wolf, T., Chaumond, J., Debut, L., Sanh, V., Delangue, C., Moi, A., Cistac, P., Funtowicz, M., Davison, J., Shleifer, S., Platen, V.P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.: Transformers: state-of-the-art natural language processing. In: Empirical Methods in Natural Language Processing (EMNLP-20) (2020). https://aclanthology.org/2020.emnlp-demos.6/

  42. World Health Organization (WHO): Anatomical Therapeutic Chemical (ATC) Toolkit. Technical Report (2021)

    Google Scholar 

  43. Xiao, C., Choi, E., Sun, J.: Opportunities & challenges in developing deep learning models using EHR data: a review. J. Am. Med. Inf. Assoc. 25(10) (2018). https://doi.org/10.1093/jamia/ocy068

Download references

Acknowledgements

This work was supported by the Natural Sciences and Engineering Research Council of Canada, Southern Ontario Smart Computing Innovation Platform, and Canadian Department of National Defence: Innovation for Defence Excellence and Security Program.

Author information

Authors and Affiliations

  1. School of Biomedical Engineering, McMaster University, Hamilton, ON, L8S 4L8, Canada

    Omar Boursalie & Thomas E. Doyle

  2. Department of Electrical, Computer, and Biomedical Engineering, Toronto Metropolitan University, Toronto, ON, M5B 2K3, Canada

    Reza Samavi

  3. Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON, L8S 4L8, Canada

    Thomas E. Doyle

  4. Vector Institute, Toronto, ON, M5G 1M1, Canada

    Omar Boursalie, Reza Samavi & Thomas E. Doyle

Authors
  1. Omar Boursalie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reza Samavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas E. Doyle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Omar Boursalie .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Kashmir, Srinagar, India

    M. Arif Wani

  2. Centre for Computational Science and Mathematical Modelling, Coventry University, Coventry, UK

    Vasile Palade

Appendices

Appendix: A

In this appendix, an overview of the dataset used in this study is provided. Tables 7 and 8 describe the types of medical data available. Next, the breakdown of records from each data source is shown in Table 9. Then, Table 10 describes the characteristics of the patient population recorded in the dataset.

Table 7 Types of medical data available in dataset
Full size table
Table 8 ICD-CA-10 diagnostic chapter codes description [7]
Full size table
Table 9 Dataset data sources breakdown
Full size table
Table 10 Characteristics of the patient population in dataset (Mean ± SE)
Full size table

Appendix: B

In this appendix, the frequency of the thirty most frequent diagnostic and visit-level THRE encoding elements for E1 (Table 11) and E2 (Table 12) is presented.

Table 11 Frequency of thirty most frequent diagnostic (left) and visit-level THRE (right) encodings elements in E1
Full size table
Table 12 Frequency of diagnostic-level (left) and thirty most frequent visit-level THRE (right) encodings elements in E2
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Boursalie, O., Samavi, R., Doyle, T.E. (2023). Evaluation of Sequential and Temporally Embedded Deep Learning Models for Health Outcome Prediction. In: Wani, M.A., Palade , V. (eds) Deep Learning Applications, Volume 4. Advances in Intelligent Systems and Computing, vol 1434. Springer, Singapore. https://doi.org/10.1007/978-981-19-6153-3_2

Download citation

Publish with us

Policies and ethics

Search

Navigation