Abstract
Evaluation of ontologies is increasingly becoming important as the number of available ontologies is steadily growing. Ontology evaluation is a labour intensive and laborious job. Hence, the importance to come up with automated methods. Before automated methods achieve reliability and widespread adoption, these methods themselves have to be assessed first by human experts. We summarise experiences acquired when trying to assess an automated ontology evaluation method. Previously we have implemented and evaluated a light-weight automatic ontology evaluation method that can be easily applied by knowledge engineers to rapidly determine whether or not the most important notions and relationships are represented in a set of ontology triplets. Domain experts have contributed to the assessment effort. Various assessment experiments have been carried out. In this paper, we focus particularly on the practical lessons learnt, in particular the limitations that result from real life constraints, rather than on the precise method to automatically evaluate results of an ontology miner. A typology of potential evaluation biases is applied to demonstrate the substantial impact conditions in which an evaluation happens can have on the reliability of the outcomes of an evaluation exercise. As a result, the notion of “meta-evaluation of ontologies” is introduced and its importance illustrated. The main conclusion is that still more domain experts have to be involved, which is exactly what we try to avoid by applying an automated evaluation procedure. A catch-22 situation?
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Alani, H., Brewster, C., Shadbolt, N.: Ranking ontologies with aktiverank. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 1–15. Springer, Heidelberg (2006)
Brank, J., Grobelnik, M., Mladenić, D.: Ontology evaluation. SEKT Deliverable #D1.6.1, Jozef Stefan Institute, Prague (2005)
Brewster, C., Alani, H., Dasmahapatra, S., Wilks, Y.: Data driven ontology evaluation. In: Shadbolt, N., O’Hara, K. (eds.) Advanced Knowledge Technologies: selected papers, pp. 164–168. AKT (2004)
Buchholz, S., Veenstra, J., Daelemans, W.: Cascaded grammatical relation assignment. In: Proceedings of EMNLP/VLC 1999, PrintPartners Ipskamp (1999)
Buitelaar, P., Cimiano, P., Loos, B. (eds.): Proceedings of the 2nd Workshop on Ontology Learning and Population: Bridging the Gap between Text and Knowledge. Association for Computational Linguistics (2006)
Buitelaar, P., Cimiano, P., Magnini, B. (eds.): Ontology Learning from Text: Methods, Applications and Evaluation. IOS Press, Amsterdam (2005)
Burton-Jones, A., Storey, V., Sugumaran, V.: A semiotic metrics suite for assessing the quality of ontologies. Data and Knowledge Engineering 55(1), 84–102 (2005)
Friedman, C., Hripcsak, G.: Evaluating natural language processors in the clinical domain. Methods of Information in Medicine 37(1-2), 334–344 (1998)
Gangemi, A., Catenacci, C., Ciaramita, M., Gil, R., Lehmann, J.: Ontology evaluation and validation: an integrated formal model for the quality diagnostic task. Technical report (2005), http://www.loa-cnr.it/Publications.html
Gillam, L., Tariq, M.: Ontology via terminology? In: Ibekwe-San Juan, F., Lainé Cruzel, S. (eds.) Proceedings of the Workshop on Terminology, Ontology and Knowledge Representation (2004), http://www.univ-lyon3.fr/partagedessavoirs/termino2004/programgb.htm
Guarino, N., Persidis, A.: Evaluation framework for content standards. OntoWeb Deliverable #D3.5, Padova (2003)
Guarino, N., Welty, C.: Evaluating ontological decisions with OntoClean. Communications of the ACM 45(2), 61–65 (2002)
Gulla, J., Borch, H., Ingvaldsen, J.: Ontology learning for search applications. In: Meersman, R., Tari, Z., et al. (eds.) OTM 2007, Part I. LNCS, vol. 4803, pp. 1050–1062. Springer, Heidelberg (2007)
Hartmann, J., Spyns, P., Maynard, D., Cuel, R., de Figueroa, S., Sure, Y.: Methods for ontology evaluation. KnowledgeWeb Deliverable #D1.2.3, 3 (2005)
Judge, J., Sogrin, M., Troussov, A.: Galaxy: IBM ontological network miner. In: CSSW. LNI, vol. 113, pp. 157–160 (2007)
Luhn, H.P.: The automatic creation of literature abstracts. IBM Journal of Research and Development 2(2), 159–195 (1958)
Maedche, A., Staab, S.: Measuring similarity between ontologies. In: Gómez-Pérez, A., Benjamins, V.R. (eds.) EKAW 2002. LNCS (LNAI), vol. 2473, pp. 251–263. Springer, Heidelberg (2002)
Maynard, D., Peters, W., Li, Y.: Evaluating evaluation metrics for ontology-based applications: Infinite reflection. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Tapias, D. (eds.) Proceedings of the Sixth International Language Resources and Evaluation (LREC 2008), Paris, European Language Resources Association (2008)
Navigli, R., Velardi, P.: Learning domain ontologies from document warehouses and dedicated web sites. Computational Linguistics 30(2), 151–179 (2004)
Noy, N., Guha, R., Musen, M.: User ratings of ontologies: who will rate the raters? In: AAAI 2005 Spring Symposium on Knowledge Collection from Volunteer Contributors (2005)
Obrst, L., Ashpole, B., Ceusters, W., Mani, I., Ray, S., Smith, B.: Semantic Web: Revolutionizing Knowledge Discovery in the Life Sciences. In: The Evaluation of Ontologies: toward Improved Semantic Interoperability, pp. 139–158. Springer, Heidelberg (2007)
Reinberger, M.-L., Spyns, P.: Unsupervised text mining for the learning of DOGMA-inspired ontologies. In: Buitelaar, Ph., Cimiano, P., Magnini, B. (eds.) Ontology Learning from Text: Methods, Applications and Evaluation, pp. 29–43. IOS Press, Amsterdam (2005)
Reinberger, M.-L., Spyns, P., Pretorius, A.J., Daelemans, W.: Automatic initiation of an ontology. In: Meersman, R., Tari, Z. (eds.) OTM 2004. LNCS, vol. 3290, pp. 600–617. Springer, Heidelberg (2004)
Sabou, M., Lopez, V., Motta, E., Uren, V.: Ontology selection: Ontology evaluation on the real semantic web. In: Proceedings of the 4th International EON Workshop, Evaluation of Ontologies for the Web (2006), http://eprints.aktors.org/487/
Sabou, M., Wroe, C., Goble, C., Mishne, G.: Learning domain ontologies for web service descriptions: an experiment in bioinformatics. In: Proceedings of the 14th International World Wide Web Conference (2005)
Shamsfard, M., Barforoush, A.: The state of the art in ontology learning: a framework for comparison. Knowledge Engineering Review 18(4), 293–316 (2003)
Siorpaes, K., Hepp, M.: Games with a purpose for the semantic web. IEEE Intelligent Systems 23(3), 50–60 (2008)
Spyns, P., Reinberger, M.-L.: Lexically evaluating ontology triples automatically generated from text. In: Gómez-Pérez, A., Euzenat, J. (eds.) ESWC 2005. LNCS, vol. 3532, pp. 563–577. Springer, Heidelberg (2005)
Spyns, P.: Evalexon: assessing triples mined from texts. Technical Report 09, STAR Lab, Brussel (2005)
Spyns, P.: Object role modelling for ontology engineering in the DOGMA framework. In: Meersman, R., Tari, Z., Herrero, P., et al. (eds.) OTM-WS 2005. LNCS, vol. 3762, pp. 710–719. Springer, Heidelberg (2005)
Spyns, P.: Validating evalexon: validating a tool for evaluating automatically lexical triples mined from texts. Technical Report x6, STAR Lab, Brussel (2005)
Spyns, P., Hogben, G.: Validating an automated evaluation procedure for ontology triples in the privacy domain. In: Moens, M.-F., Spyns, P. (eds.) Proceedings of the 18th Annual Conference on Legal Knowledge and Information Systems (JURIX 2005), pp. 127–136. IOS Press, Amsterdam (2005)
Spyns, P., Pretorius, A.J., Reinberger, M.-L.: Evaluating DOGMA-lexons generated automatically from a text. In: Cimiano, P., Ciravegna, F., Motta, E., Uren, V. (eds.) EKAW 2004. LNCS (LNAI), vol. 3257, pp. 38–44. Springer, Heidelberg (2004)
Stvilia, B.: A model for ontology quality evaluation. First Monday 12(12) (2007)
Thompson Automation Software, Jefferson OR, US. Tawk Compiler, v.5 edition
Velardi, P., Missikoff, M., Basili, R.: Identification of relevant terms to support the construction of domain. In: Maybury, M., Bernsen, N., Krauwer, S. (eds.) Proc. of the ACL-EACL Workshop on Human Language Technologies (2001)
Vossen, P., Agirre, E., Calzolari, N., et al.: Kyoto: a system for mining, structuring and distributing knowledge across languages and cultures. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Tapias, D. (eds.) Proceedings of the Sixth International Language Resources and Evaluation (LREC 2008), Paris, European Language Resources Association (2008)
Vrandecic, D., Sure, Y.: How to design better ontology metrics. In: May, W., Kifer, M. (eds.) ESWC 2007. LNCS, vol. 4519, pp. 311–325. Springer, Heidelberg (2007)
Zipf, G.K.: Human Behaviour and the Principle of Least-Effort. Addison-Wesley, Cambridge (1949)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Spyns, P. (2008). Evaluating Automatically a Text Miner for Ontologies: A Catch-22 Situation?. In: Meersman, R., Tari, Z. (eds) On the Move to Meaningful Internet Systems: OTM 2008. OTM 2008. Lecture Notes in Computer Science, vol 5332. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88873-4_33
Download citation
DOI: https://doi.org/10.1007/978-3-540-88873-4_33
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-88872-7
Online ISBN: 978-3-540-88873-4
eBook Packages: Computer ScienceComputer Science (R0)