Abstract
Evaluating discovery systems is a fundamentally challenging task because if they are successful, by definition they are capturing new knowledge that has yet to be proven useful. To overcome this difficulty, many researchers in literature-based discovery (LBD) replicated Swanson's discoveries to evaluate the performance of their systems. They reported overall success if one of the discoveries generated by their system was the same as Swanson's discovery. This type of evaluation is powerful yet incomplete because it does not inform us about the quality of the rest of the discoveries identified by the system nor does it test the generalizability of the results. Recently, alternative evaluation methods have been designed to provide more information on the overall performance of the systems. The purpose of this chapter is to review and analyze the current evaluation methods for LBD systems and to discuss potential ways to use these evaluation methods for comparing performance of different systems, rather than reporting the performance of only one system. We will also summarize the current approaches used to evaluate the graphical user interfaces of LBD systems.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Baeza-Yates, R. and Ribeiro-Neto, B., Modern Information Retrieval. 1999: ACM Press, Addison-Wesley
Bradley, A.P., The Use of the Area Under the Roc Curve in the Evaluation of Machine Learning Algorithms. Pattern Recognit., 1997. 30(7): 1145–1159
Swanson, D.R., Migraine and Magnesium: Eleven Neglected Connections. Perspect. Biol. Med., 1988. 31: 526–557
Swanson, D.R., Fish Oil, Raynaud’s Syndrome, and Undiscovered Public Knowledge. Perspect. Biol. Med., 1986. 30(1): 7–15
Smalheiser, N.R. and Swanson, D., Linking Estrogen to Alzheimer’s Disease: An Informatics Approach. Neurology, 1996. 47(3): 809–810
Smalheiser, N.R. and Swanson, D., Indomethacin and Alzheimer’s Disease. Neurology, 1996. 46(2): 583
Swanson, D.R., Somatomedin C and Arginine: Implicit Connections Between Mutually Isolated Literatures. Perspect. Biol. Med., 1990. 33(2): 157–186
Smalheiser, N.R. and Swanson, D., Calcium-Independent Phospholipase A2 and Schizophrenia. Arch. Gen. Psychiatry, 1998. 55: 752–753
Lindsay, R.K. and Gordon, M.D., Literature Based Discovery by Lexical Statistics. J. Am. Soc. Inf. Sci., 1999. 49(8): 674–685
Gordon, M.D. and Dumais, S., Using Latent Semantic Indexing for Literature Based Discovery. J. Am. Soc. Inf. Sci., 1998. 49(8): 674–685
Blake, C. and Pratt, W., Automatically Identifying Candidate Treatments from Existing Medical Literature. In Proceedings of AAAI Spring Symposium on Mining Answers from Texts and Knowledge Bases. 2002. Stanford, CA
Weeber, M., Klein, H., and de Jong-van den Berg, L.T.W., Using Concepts in Literature Based Discovery: Simulating Swanson’s Raynaud–Fish Oil and Migraine–Magnesium Examples. J. Am. Soc. Inf. Sci., 2001. 52(7): 548–557
Srinivasan, P., Generating Hypotheses from MEDLINE. J. Am. Soc. Inf. Sci., 2004. 55(5): 396–413
Hu, X., Li, G., Yoo, I., Zhang, X., and Xu, X., A Semantic-Based Approach for Mining Undiscovered Public Knowledge from Biomedical Knowledge. In Proceedings of IEEE International Conference on Granular Computing. 2005. Beijing
Pratt, W. and Yetisgen-Yildiz, M. L., Capturing Connections Across the Biomedical Literature. In Proceedings of International Conference on Knowledge Capture (K-Cap’03). 2003. Florida
Hristovski, D., Stare, J., Peterlin, B., and Dzeroski, S., Supporting Discovery in Medicine by Association Rule Mining in Medline and UMLS. In Proceedings of Medinfo Conference. 2001. London, England
Yetisgen-Yildiz, M. and Pratt, W., Using Statistical and Knowledge-Based Approaches for Literature Based Discovery. J. Biomed. Inform., 2006. 39(6): 600–611
Wren, J.D., Extending the Mutual Information Measure to Rank Inferred Literature Relationship. BMC Bioinformatics, 2004. 5(1): 145
Weeber, M., Vos, R., Klein, H., de Jong-van den Berg, L.T.W., and Aronson, A.R., Generating Hypotheses by Discovering Implicit Associations in the Literature: A Case Report of a Search for New Potential Therapeutic Uses for Thalidomide. J. Am. Med. Inform. Assoc., 2003. 10(3): 252–259
Wren, J.D., Bekeredjian, R., Stewart, J.A., Shohet, R.V., and Garner, H.R., Knowledge Discovery by Automated Identification and Ranking of Implicit Relationships. Bioinformatics, 2004. 20(3): 389–398
Swanson, D.R., Atrial Fibrillation in Athletes: Implicit Literature-Based Connections Suggest that Overtraining and Subsequent Inflammation may be a Contributory Mechanism. Med. Hypotheses, 2006. 66(6): 1085–1092
Skeels, M.M., Henning, K., Yetisgen-Yildiz, M., and Pratt, W., Interaction Design for Literature-Based Discovery. In Proceedings of the International Conference for Human-Computer Interaction (CHI’05). 2005. Portland, WA
Smalheiser, N.R., et al., Collaborative Development of the Arrowsmith Two Node Search Interface Designed for Laboratory Investigators. J. Biomed. Discov. Collab., 2006. 1(8)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Yetisgen-Yildiz, M., Pratt, W. (2008). Evaluation of Literature-Based Discovery Systems. In: Bruza, P., Weeber, M. (eds) Literature-based Discovery. Information Science and Knowledge Management, vol 15. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68690-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-540-68690-3_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68685-9
Online ISBN: 978-3-540-68690-3
eBook Packages: Computer ScienceComputer Science (R0)