Skip to main content
SpringerLink
Log in

FATE: Annotating a Textual Entailment Corpus with FrameNet

  • Chapter
  • First Online:
Handbook of Linguistic Annotation

Abstract

Several works show that predicate-argument structure is a level of analysis relevant for addressing Natural Language Processing problems, such as Textual Entailment (another study on Textual Entailment can be found in this volume). Although large resources like FrameNet are available (see also the chapter on FrameNet in this volume), attempts to integrate this type of information into a system for textual entailment has not delivered the expected gain in performance. The reasons for this result are not fully obvious; candidates include FrameNet’s restricted coverage, limitations of semantic parsers, or insufficient modeling of FrameNet information. To enable further insight on this issue, in this paper we present FATE (FrameNet-Annotated Textual Entailment), a manually built, fully reliable frame-annotated RTE corpus. The annotation covers the 800 pairs of the RTE-2 test set. This dataset offers a safe basis for RTE systems to experiment, and enables researchers to develop clearer ideas on how to integrate frame knowledge effectively into semantic inference tasks like recognizing textual entailment. We describe and present statistics over the adopted annotation, which introduces a new schema based on full-text annotation of so called relevant frame-evoking elements. (This chapter is based on Burchardt, Pennacchiotti, Proceedings of the sixth international conference on language resources and evaluation (LREC’08) (2008) [7].)

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 349.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 449.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 449.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    See, e.g., [16]. For the same reason, PropBank’s Arg2...ArgN roles are not generalizable [23].

  2. 2.

    http://framenet.icsi.berkeley.edu.

  3. 3.

    The noun and adjective/adverb more evoke the frame Increment.

  4. 4.

    Three more guidelines better specify the definition: (1) cases in which all role fillers are self-references to the FEE must be considered non relevant; (2) in the case that a candidate relevant FEE evokes a situation which is not represented as a frame in FrameNet, the annotator can evoke a special unknown frame; (3) a relevant FEE can be either a single word or a multiword expression.

  5. 5.

    Salto can be obtained from http://www.coli.uni-saarland.de/projects/salsa/page.php?id=software.

  6. 6.

    More particularly, for each annotator we divide the number of FEE by the number FEE shared with the other annotator in order to compute FEE-agreement. Then we compute the average. The values for each of these are calculated as follows:

    1. a.

      To compute frame-agreement, for each annotator we consider the frames which have been evoked by an FEE shared with the other annotator. Then we compute the percentage of those frames that have been evoked also by the other annotator. Finally, we compute the percentage average between the two annotators.

    2. b.

      To compute role-agreement we consider only the roles belonging to frames in common between the annotators (same evoking FEE and same frame name). Then we compute the percentage of these roles that have the same name and the same lexical fillers.

    3. c.

      Finally, we compute the percentage average between the two annotators.

    The obtained agreements are: 82% FEE-agreement, 88% frame-agreement, 91% role-agreement. These results indicate that the overall annotation is reliable. In particular, our definition of relevant FEE seems to be plausible and effective, as the two annotators selected the same FEEs in 82% of cases. Also, once the FEE has been selected, the tasks of finding the correct frame and the correct roles seems to be fairly easy and unambiguous. The sporadic cases of disagreement on frames usually involve the choice of different but highly similar frames (e.g. Risky_situation vs. Run_risk) or an unknown frame used by one annotator instead of the correct one present in the FrameNet hierarchy. Cases of disagreements on roles are generally due by one annotator missing a role.

References

  1. Baker, C.F., Fillmore, C.J., Lowe, J.B.: The Berkeley FrameNet project. In: Proceedings of COLING-ACL, Canada (1998)

    Google Scholar 

  2. Bar-Haim, R., Szpektor, I., Glickman, O.: Definition and analysis of intermediate entailment levels. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pp. 55–60. Ann Arbor, Michigan (2005)

    Google Scholar 

  3. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., Szpektor, I. (eds.): In: Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Italy (2006)

    Google Scholar 

  4. Bentivogli, L., Clark, P., Dagan, I., Dang, H., Giampiccolo, D.: The seventh pascal recognizing textual entailment challenge. In: Proceedings of the Text Analytic Conference (TAC 2011), Gaithersburg (2011)

    Google Scholar 

  5. Bos, J., Markert, K.: Combining shallow and deep NLP methods for recognizing textual entailment. In: Pascal, Proceedings of the First Challenge Workshop, Recognizing Textual Entailment, Southampton (2005)

    Google Scholar 

  6. Burchardt, A., Frank, A.: Approximating textual entailment with LFG and FrameNet frames. In: Proceedings of PASCAL RTE2 Workshop (2006)

    Google Scholar 

  7. Burchardt, A., Pennacchiotti, M.: FATE: a FrameNet-annotated corpus for textual entailment. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Tapias, D. (eds.) Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). European Language Resources Association (ELRA), Morocco (2008)

    Google Scholar 

  8. Burchardt, A., Erk, K., Frank, A.: A WordNet detour to FrameNet. In: Fisseni, B., Schmitz, H.C., Schröder, B., Wagner, P. (eds.) Sprachtechnologie, Mobile Kommunikation und Linguistische Resourcen, Computer Studies in Language and Speech, vol. 8. Peter Lang, Frankfurt (2005)

    Google Scholar 

  9. Burchardt, A., Erk, K., Frank, A., Kowalski, A., Pado, S., Pinkal, M.: The salsa corpus: a german corpus resource for lexical semantics. In: Proceedings of LREC 2006, Italy (2006a)

    Google Scholar 

  10. Burchardt, A., Erk, K., Frank, A., Kowalski, A., Pado, S., Pinkal, M.: Salto – a versatile multi-level annotation tool. In: Proceedings of LREC 2006, Italy (2006b)

    Google Scholar 

  11. Burchardt, A., Reiter, N., Thater, S., Frank, A.: A semantic approach to textual entailment: system evaluation and task analysis. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague (2007)

    Google Scholar 

  12. Burchardt, A., Pennacchiotti, M., Thater, S., Pinkal, M.: Assessing the impact of frame semantics on textual entailment. Nat. Lang. Eng. 15(4), 527–550 (2009)

    Article  Google Scholar 

  13. Collins, M.: Head-driven statistical models for natural language parsing. Ph.D. Thesis, University of Pennsylvania, Philadelphia (1999)

    Google Scholar 

  14. Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., D’Alché-Buc, F. (eds.) Evaluating Predictive Uncertainty, Visual Object Categorization and Textual Entailment. Lecture Notes in Computer Science, vol. 3944, pp. 1–27. Springer, Heidelberg (2006)

    Google Scholar 

  15. Erk, K., Pado, S.: Shalmaneser - a flexible toolbox for semantic role assignment. In: Proceedings of LREC 2006, Italy (2006)

    Google Scholar 

  16. Fillmore, C.J., Baker, C.: A frames approach to semantic analysis. In: Heine, B., Narrog, H. (eds.) The Oxford Handbook of Linguistic Analysis, pp. 313–339. Oxford University Press, Oxford (2010)

    Google Scholar 

  17. Garoufi, K.: Towards a better understanding of applied textual entailment: annotation and evaluation of the RTE-2 dataset. M.Sc. Thesis, Saarland University (2007)

    Google Scholar 

  18. Kingsbury, P., Palmer, M., Marcus, M.: Adding semantic annotation to the Penn TreeBank. In: Proceedings of the Human Language Technology Conference, San Diego (2002)

    Google Scholar 

  19. Litkowski, K.: Componential analysis for recognizing textual entailment. In: Proceedings of PASCAL RTE2 Workshop (2006)

    Google Scholar 

  20. Ovchinnikova, E., Vieu, L., Oltramari, A., Borgo, S., Alexandrov, T.: Data-driven and ontological analysis of framenet for natural language reasoning. In: Chair, N.C.C., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., Tapias, D. (eds.) Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Malta (2010)

    Google Scholar 

  21. Ovchinnikova, E., Hobbs, J.R., Montazeri, N., McCord, M.C., Alexandrov, T., Mulkar-Mehta, R.: Abductive reasoning with a large knowledge base for discourse processing. In: Proceedings of the Ninth International Conference on Computational Semantics, Association for Computational Linguistics, Stroudsburg, IWCS ’11, pp. 225–234 (2011)

    Google Scholar 

  22. Pado, S.: Cross-lingual annotation projection models for role-semantic information. Ph.D. Thesis, Saarland University, Germany (2007)

    Google Scholar 

  23. Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: an annotated corpus of semantic roles. Comput. Linguist. 31(1), 71–106 (2005). doi:10.1162/0891201053630264

    Article  Google Scholar 

  24. Ruppenhofer, J., Ellsworth, M., Petruck, M.R.L., Johnson, C.R.: FrameNet: theory and practice (2007). http://framenet.icsi.berkeley.edu/

  25. Ruppenhofer, J., Sunde, J., Pinkal, M.: Generating FrameNets of various granularities: The FrameNet transformer. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., Tapias, D. (eds.) Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Malta (2010)

    Google Scholar 

  26. Sekine, S., Inui, K., Dagan, I., Dolan, B., Giampiccolo, D., Magnini, B. (eds.): Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics, Prague (2007)

    Google Scholar 

  27. Vanderwende, L., Menezes, A., Snow, R.: Microsoft Research at RTE-2: Syntactic contributions in the entailment task: an implementation. In: Magnini, B., Dagan, I. (eds.) Proceedings of the Second PASCAL Recognizing Textual Entailment Challenge. Springer, Italy (2006)

    Google Scholar 

Download references

Acknowledgements

Thanks to Konstantina Garoufi for providing the span annotation and to Alexander Fleisch for leading the annotation work. Thanks a lot to the anonymous reviewers for valuable comments and corrections. This work has partly been funded by the German Research Foundation DFG (grant PI 154/9-3).

Author information

Authors and Affiliations

  1. DFKI, Language Technology Lab, Alt-Moabit 91c, 10559, Berlin, Germany

    Aljoscha Burchardt

  2. EBay Inc., 2065 Hamilton Ave, San Jose, CA, 95125, USA

    Marco Pennacchiotti

Authors
  1. Aljoscha Burchardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Pennacchiotti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aljoscha Burchardt .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Vassar College, Poughkeepsie, New York, USA

    Nancy Ide

  2. Department of Computer Science, Volen Center for Complex Systems, Brandeis University, Waltham, Massachusetts, USA

    James Pustejovsky

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Burchardt, A., Pennacchiotti, M. (2017). FATE: Annotating a Textual Entailment Corpus with FrameNet. In: Ide, N., Pustejovsky, J. (eds) Handbook of Linguistic Annotation. Springer, Dordrecht. https://doi.org/10.1007/978-94-024-0881-2_41

Download citation

  • DOI: https://doi.org/10.1007/978-94-024-0881-2_41

  • Published:

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-024-0879-9

  • Online ISBN: 978-94-024-0881-2

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation