Abstract
Several works show that predicate-argument structure is a level of analysis relevant for addressing Natural Language Processing problems, such as Textual Entailment (another study on Textual Entailment can be found in this volume). Although large resources like FrameNet are available (see also the chapter on FrameNet in this volume), attempts to integrate this type of information into a system for textual entailment has not delivered the expected gain in performance. The reasons for this result are not fully obvious; candidates include FrameNet’s restricted coverage, limitations of semantic parsers, or insufficient modeling of FrameNet information. To enable further insight on this issue, in this paper we present FATE (FrameNet-Annotated Textual Entailment), a manually built, fully reliable frame-annotated RTE corpus. The annotation covers the 800 pairs of the RTE-2 test set. This dataset offers a safe basis for RTE systems to experiment, and enables researchers to develop clearer ideas on how to integrate frame knowledge effectively into semantic inference tasks like recognizing textual entailment. We describe and present statistics over the adopted annotation, which introduces a new schema based on full-text annotation of so called relevant frame-evoking elements. (This chapter is based on Burchardt, Pennacchiotti, Proceedings of the sixth international conference on language resources and evaluation (LREC’08) (2008) [7].)
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
The noun and adjective/adverb more evoke the frame Increment.
- 4.
Three more guidelines better specify the definition: (1) cases in which all role fillers are self-references to the FEE must be considered non relevant; (2) in the case that a candidate relevant FEE evokes a situation which is not represented as a frame in FrameNet, the annotator can evoke a special unknown frame; (3) a relevant FEE can be either a single word or a multiword expression.
- 5.
Salto can be obtained from http://www.coli.uni-saarland.de/projects/salsa/page.php?id=software.
- 6.
More particularly, for each annotator we divide the number of FEE by the number FEE shared with the other annotator in order to compute FEE-agreement. Then we compute the average. The values for each of these are calculated as follows:
-
a.
To compute frame-agreement, for each annotator we consider the frames which have been evoked by an FEE shared with the other annotator. Then we compute the percentage of those frames that have been evoked also by the other annotator. Finally, we compute the percentage average between the two annotators.
-
b.
To compute role-agreement we consider only the roles belonging to frames in common between the annotators (same evoking FEE and same frame name). Then we compute the percentage of these roles that have the same name and the same lexical fillers.
-
c.
Finally, we compute the percentage average between the two annotators.
The obtained agreements are: 82% FEE-agreement, 88% frame-agreement, 91% role-agreement. These results indicate that the overall annotation is reliable. In particular, our definition of relevant FEE seems to be plausible and effective, as the two annotators selected the same FEEs in 82% of cases. Also, once the FEE has been selected, the tasks of finding the correct frame and the correct roles seems to be fairly easy and unambiguous. The sporadic cases of disagreement on frames usually involve the choice of different but highly similar frames (e.g. Risky_situation vs. Run_risk) or an unknown frame used by one annotator instead of the correct one present in the FrameNet hierarchy. Cases of disagreements on roles are generally due by one annotator missing a role.
-
a.
References
Baker, C.F., Fillmore, C.J., Lowe, J.B.: The Berkeley FrameNet project. In: Proceedings of COLING-ACL, Canada (1998)
Bar-Haim, R., Szpektor, I., Glickman, O.: Definition and analysis of intermediate entailment levels. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pp. 55–60. Ann Arbor, Michigan (2005)
Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., Szpektor, I. (eds.): In: Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Italy (2006)
Bentivogli, L., Clark, P., Dagan, I., Dang, H., Giampiccolo, D.: The seventh pascal recognizing textual entailment challenge. In: Proceedings of the Text Analytic Conference (TAC 2011), Gaithersburg (2011)
Bos, J., Markert, K.: Combining shallow and deep NLP methods for recognizing textual entailment. In: Pascal, Proceedings of the First Challenge Workshop, Recognizing Textual Entailment, Southampton (2005)
Burchardt, A., Frank, A.: Approximating textual entailment with LFG and FrameNet frames. In: Proceedings of PASCAL RTE2 Workshop (2006)
Burchardt, A., Pennacchiotti, M.: FATE: a FrameNet-annotated corpus for textual entailment. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Tapias, D. (eds.) Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). European Language Resources Association (ELRA), Morocco (2008)
Burchardt, A., Erk, K., Frank, A.: A WordNet detour to FrameNet. In: Fisseni, B., Schmitz, H.C., Schröder, B., Wagner, P. (eds.) Sprachtechnologie, Mobile Kommunikation und Linguistische Resourcen, Computer Studies in Language and Speech, vol. 8. Peter Lang, Frankfurt (2005)
Burchardt, A., Erk, K., Frank, A., Kowalski, A., Pado, S., Pinkal, M.: The salsa corpus: a german corpus resource for lexical semantics. In: Proceedings of LREC 2006, Italy (2006a)
Burchardt, A., Erk, K., Frank, A., Kowalski, A., Pado, S., Pinkal, M.: Salto – a versatile multi-level annotation tool. In: Proceedings of LREC 2006, Italy (2006b)
Burchardt, A., Reiter, N., Thater, S., Frank, A.: A semantic approach to textual entailment: system evaluation and task analysis. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague (2007)
Burchardt, A., Pennacchiotti, M., Thater, S., Pinkal, M.: Assessing the impact of frame semantics on textual entailment. Nat. Lang. Eng. 15(4), 527–550 (2009)
Collins, M.: Head-driven statistical models for natural language parsing. Ph.D. Thesis, University of Pennsylvania, Philadelphia (1999)
Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., D’Alché-Buc, F. (eds.) Evaluating Predictive Uncertainty, Visual Object Categorization and Textual Entailment. Lecture Notes in Computer Science, vol. 3944, pp. 1–27. Springer, Heidelberg (2006)
Erk, K., Pado, S.: Shalmaneser - a flexible toolbox for semantic role assignment. In: Proceedings of LREC 2006, Italy (2006)
Fillmore, C.J., Baker, C.: A frames approach to semantic analysis. In: Heine, B., Narrog, H. (eds.) The Oxford Handbook of Linguistic Analysis, pp. 313–339. Oxford University Press, Oxford (2010)
Garoufi, K.: Towards a better understanding of applied textual entailment: annotation and evaluation of the RTE-2 dataset. M.Sc. Thesis, Saarland University (2007)
Kingsbury, P., Palmer, M., Marcus, M.: Adding semantic annotation to the Penn TreeBank. In: Proceedings of the Human Language Technology Conference, San Diego (2002)
Litkowski, K.: Componential analysis for recognizing textual entailment. In: Proceedings of PASCAL RTE2 Workshop (2006)
Ovchinnikova, E., Vieu, L., Oltramari, A., Borgo, S., Alexandrov, T.: Data-driven and ontological analysis of framenet for natural language reasoning. In: Chair, N.C.C., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., Tapias, D. (eds.) Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Malta (2010)
Ovchinnikova, E., Hobbs, J.R., Montazeri, N., McCord, M.C., Alexandrov, T., Mulkar-Mehta, R.: Abductive reasoning with a large knowledge base for discourse processing. In: Proceedings of the Ninth International Conference on Computational Semantics, Association for Computational Linguistics, Stroudsburg, IWCS ’11, pp. 225–234 (2011)
Pado, S.: Cross-lingual annotation projection models for role-semantic information. Ph.D. Thesis, Saarland University, Germany (2007)
Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: an annotated corpus of semantic roles. Comput. Linguist. 31(1), 71–106 (2005). doi:10.1162/0891201053630264
Ruppenhofer, J., Ellsworth, M., Petruck, M.R.L., Johnson, C.R.: FrameNet: theory and practice (2007). http://framenet.icsi.berkeley.edu/
Ruppenhofer, J., Sunde, J., Pinkal, M.: Generating FrameNets of various granularities: The FrameNet transformer. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., Tapias, D. (eds.) Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Malta (2010)
Sekine, S., Inui, K., Dagan, I., Dolan, B., Giampiccolo, D., Magnini, B. (eds.): Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics, Prague (2007)
Vanderwende, L., Menezes, A., Snow, R.: Microsoft Research at RTE-2: Syntactic contributions in the entailment task: an implementation. In: Magnini, B., Dagan, I. (eds.) Proceedings of the Second PASCAL Recognizing Textual Entailment Challenge. Springer, Italy (2006)
Acknowledgements
Thanks to Konstantina Garoufi for providing the span annotation and to Alexander Fleisch for leading the annotation work. Thanks a lot to the anonymous reviewers for valuable comments and corrections. This work has partly been funded by the German Research Foundation DFG (grant PI 154/9-3).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Burchardt, A., Pennacchiotti, M. (2017). FATE: Annotating a Textual Entailment Corpus with FrameNet. In: Ide, N., Pustejovsky, J. (eds) Handbook of Linguistic Annotation. Springer, Dordrecht. https://doi.org/10.1007/978-94-024-0881-2_41
Download citation
DOI: https://doi.org/10.1007/978-94-024-0881-2_41
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-024-0879-9
Online ISBN: 978-94-024-0881-2
eBook Packages: Social SciencesSocial Sciences (R0)