Abstract
Precision and Recall, as well as their combination in terms of F-Measure, are widely used measures in computer science and generally applied to evaluate the overall performance of ontology matchers in fully automatic, unsupervised scenarios. In this paper, we investigate the case of supervised matching, where automatically created ontology alignments are verified by an expert. We motivate and describe this use case and its characteristics and discuss why traditional, F-measure based evaluation measures are not suitable for this use case. Therefore, we investigate several alternative evaluation measures and propose the use of Precision@N curves as a mean to assess different matching systems for supervised matching. We compare the ranking of several state of the art matchers using Precision@N curves to the traditional F-measure based ranking, and discuss means to combine matchers in a way that optimizes the user support in supervised ontology matching.
Chapter PDF
Similar content being viewed by others
Keywords
References
Aguirre, J.L., Eckert, K., Euzenat, J., Ferrara, A., van Hage, W.R., Hollink, L., Meilicke, C., Nikolov, A., Ritze, D., Scharffe, F., Shvaiko, P., Šváb-Zamazal, O., Trojahn, C., Jiménez-Ruiz, E., Cuenca Grau, B., Zapilko, B.: Results of the Ontology Alignment Evaluation Initiative 2012. In: Proc. of the 7th Int. Workshop on Ontology Matching (2012)
Ba, M., Diallo, G.: ServOMap and ServOMap-lt Results for OAEI 2012. In: Proc. of the 7th ISWC Workshop on Ontology Matching, OM (2012) W.R Hage, L. Hollink, C. Meilicke, A. Nikolov, D. Ritze, F. Scharffe, P. Shvaiko, O.Š. Zamazal, C. Trojahn, E. Jim/’enez-Ruiz, B.C. Grau, and B. Zapilko
Cheatham, M.: MapSSS Results for OAEI 2011. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)
Dang, T.T., Gabriel, A., Hertling, S., Roskosch, P., Wlotzka, M., Zilke, J.R., Janssen, F., Paulheim, H.: HotMatch Results for OEAI 2012. In: Proc.of the 7th ISWC Workshop on Ontology Matching, OM (2012)
David, J., Guillet, F., Briand, H.: Association rule ontology matching approach. International Journal on Semantic Web and Information Systems 3(2), 27–49 (2007)
Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. In: Proc. of the 23rd Int. Conference on Machine Learning, ICML (2006)
Duan, S., Fokoue, A., Srinivas, K.: One Size Does Not Fit All: Customizing Ontology Alignment Using User Feedback. In: Patel-Schneider, P.F., Pan, Y., Hitzler, P., Mika, P., Zhang, L., Pan, J.Z., Horrocks, I., Glimm, B. (eds.) ISWC 2010, Part I. LNCS, vol. 6496, pp. 177–192. Springer, Heidelberg (2010)
Eckert, K., Meilicke, C., Stuckenschmidt, H.: Improving Ontology Matching using Meta-level Learning. In: Aroyo, L., et al. (eds.) ESWC 2009. LNCS, vol. 5554, pp. 158–172. Springer, Heidelberg (2009)
Ehrig, M., Staab, S., Sure, Y.: Bootstrapping Ontology Alignment Methods with APFEL. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (eds.) ISWC 2005. LNCS, vol. 3729, pp. 186–200. Springer, Heidelberg (2005)
Euzenat, J., Shvaiko, P.: Ontology Matching. Springer (2007)
Fangmeyer, H.: Semi Automatic Indexing: State of the Art. Technical report, North Atlantic Treaty Organization. Advisory Group for Aerospace Research and Development (1974)
Gay, C.W., Kayaalp, M., Aronson, A.R.: Semi-Automatic Indexing of Full Text Biomedical Articles. In: Proc. of the Fall Symposium of the American Medical Informatics Association (2005)
Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29–36 (1982)
Hazewinkel, M.: Encyclopedia of Mathematics. Harmonic series. Springer (2001)
Hertling, S.: Hertuda Results for OEAI 2012. In: Proc. of the 7th ISWC Workshop on Ontology Matching, OM (2012)
Huber, J., Sztyler, T., Noessner, J., Meilicke, C.: CODI: Combinatorial Optimization for Data Integration - Results for OAEI 2011. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)
Jiménez-Ruiz, E., Grau, B.C., Zhou, Y., Horrocks, I.: Large-scale interactive ontology matching: Algorithms and implementation. In: Proc. of the 20th European Conference on Artificial Intelligence, ECAI (2012)
Kirsten, T., Gross, A., Hartung, M., Rahm, E.: Gomma: a component-based infrastructure for managing and analyzing life science ontologies and their evolution. Journal of Biomedical Semantics 2(6) (2011)
Manning, C., Schütze, H.: Foundations of statistical natural language processing. MIT Press (1999)
Mayr, P., Petras, V.: Building a Terminology Network for Search: The KoMoHe Project. In: Proc. of the Int. Conference on Dublin Core and Metadata Applications (2008)
Ngo, D., Bellahsene, Z.: YAM++: A Multi-strategy Based Approach for Ontology Matching Task. In: ten Teije, A., Völker, J., Handschuh, S., Stuckenschmidt, H., d’Acquin, M., Nikolov, A., Aussenac-Gilles, N., Hernandez, N. (eds.) EKAW 2012. LNCS, vol. 7603, pp. 421–425. Springer, Heidelberg (2012)
Ngomo, A.-C.N., Jens, L., Auer, S., Höffner, K.: RAVEN - active learning of link specifications. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)
Niu, X., Wang, H., Wu, G., Qi, G., Yu, Y.: Evaluating the Stability and Credibility of Ontology Matching Methods. In: Antoniou, G., Grobelnik, M., Simperl, E., Parsia, B., Plexousakis, D., De Leenheer, P., Pan, J. (eds.) ESWC 2011, Part I. LNCS, vol. 6643, pp. 275–289. Springer, Heidelberg (2011)
Noy, N.F., Musen, M.A.: The PROMPT suite: interactive tools for ontology merging and mapping. Int. Journal of Human-Computer Studies 59(6), 983–1024 (2003)
Paulheim, H.: WeSeE-Match Results for OEAI 2012. In: Proc. of the 7th ISWC Workshop on Ontology Matching, OM (2012)
Paulheim, H., Hertling, S., Ritze, D.: Towards evaluating interactive ontology matching tools. In: Cimiano, P., Corcho, O., Presutti, V., Hollink, L., Rudolph, S. (eds.) ESWC 2013. LNCS, vol. 7882, pp. 31–45. Springer, Heidelberg (2013)
Ritze, D., Paulheim, H.: Towards an Automatic Parameterization of Ontology Matching Tools based on Example Mappings. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)
Thomas Jr., G.B., Finney, R.L.: Calculus and Analytic Geometry, 9th edn. Addison Wesley (1996)
To, H.-V., Ichise, R., Le, H.-B.: An Adaptive Machine Learning Framework with User Interaction for Ontology Matching. In: Proc. of the International Joint Conferences on Artifical Intelligence, Workshop on Information Integration on the Web, pp. 35–40 (2009)
van Rijsbergen, C.J.: Information retrieval. Butterworths, London (1975)
Zhou, H., Kang, J., Chen, F., Yang, H.: OPTIMA: An Ontology-Based PlaTform-specIfic software Migration Approach. In: Proc. of the 7th Int. Conference on Quality Software (QSIC), pp. 143–152 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ritze, D., Paulheim, H., Eckert, K. (2013). Evaluation Measures for Ontology Matchers in Supervised Matching Scenarios. In: Alani, H., et al. The Semantic Web – ISWC 2013. ISWC 2013. Lecture Notes in Computer Science, vol 8219. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41338-4_25
Download citation
DOI: https://doi.org/10.1007/978-3-642-41338-4_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-41337-7
Online ISBN: 978-3-642-41338-4
eBook Packages: Computer ScienceComputer Science (R0)