Abstract
Distributional semantic models provide vector representations for words by gathering co-occurrence frequencies from corpora of text. Compositional distributional models extend these from words to phrases and sentences. In categorical compositional distributional semantics, phrase and sentence representations are functions of their grammatical structure and representations of the words therein. In this setting, grammatical structures are formalised by morphisms of a compact closed category and meanings of words are formalised by objects of the same category. These can be instantiated in the form of vectors or density matrices. This paper concerns the applications of this model to phrase and sentence level entailment. We argue that entropy-based distances of vectors and density matrices provide a good candidate to measure word-level entailment, show the advantage of density matrices over vectors for word level entailments, and prove that these distances extend compositionally from words to phrases and sentences. We exemplify our theoretical constructions on real data and a toy entailment dataset and provide preliminary experimental evidence.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Balkır, E.: Using Density Matrices in a Compositional Distributional Model of Meaning. Master’s thesis, University of Oxford (2014)
Balkır, E., Sadrzadeh, M., Coecke, B.: Distributional sentence entailment using density matrices. In: FTP-ENTC Proceedings of the First International Conference on Theoretical Topics in Computer Science (TTCS), vol. 9541, pp. 1–22 (2015)
Balkir, E., Kartsaklis, D., Sadrzadeh, M.: Sentence entailment in compositional distributional semantics. In: Fourteenth International Symposium on Artificial Intelligence and Mathematics. arXiv:1512.04419 (2016)
Bankova, D., Coecke, B., Lewis, M., Marsden, D.: Graded entailment for compositional distributional semantics, arXiv:1601.04908 (2016)
Baroni, M., Zamparelli, R.: Nouns are vectors, adjectives are matrices: representing adjective-noun constructions in semantic space. In: Conference on Empirical Methods in Natural Language Processing (EMNLP-10). Cambridge (2010)
Baroni, M., Bernardi, R., Do, N.Q., Shan, C.C.: Entailment above the word level in distributional semantics. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 23–32. Association for Computational Linguistics (2012)
Blacoe, W., Kashefi, E., Lapata, M.: A quantum-theoretic approach to distributional semantics. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 847–857 (2013)
Clark, S., Pulman, S.: Combining symbolic and distributional models of meaning. In: Proceedings of the AAAI Spring Symposium on Quantum Interaction, pp. 52–55 (2007)
Clarke, D.: Context-theoretic semantics for natural language: an overview. In: Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pp. 112–119. Association for Computational Linguistics (2009)
Coecke, B., Paquette, E.O.: Categories for the practising physicist. Springer, Berlin (2010)
Coecke, B., Sadrzadeh, M., Clark, S.: Mathematical foundations for a compositional distributional model of meaning. Linguist. Anal. 36, 345–384 (2010)
Dagan, I., Lee, L., Pereira, F.C.N.: Similarity-based models of word cooccurrence probabilities. Mach. Learn. 34(1–3), 43–69 (1999)
Dagan, I., Glickman, O., Magnini, B.: The pascal recognising textual entailment challenge. In: Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 177–190. Springer, Berlin (2006)
Firth, J.R.: A Synopsis of Linguistic Theory, 1930–1955. Studies in Linguistic Analysis, pp. 1–32 (1957)
Grefenstette, E., Sadrzadeh, M.: Experimental support for a categorical compositional distributional model of meaning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1394–1404. Association for Computational Linguistics (2011)
Grefenstette, E., Sadrzadeh, M.: Concrete models and empirical evaluations for the categorical compositional distributional model of meaning. Comput. Linguist. 41, 71–118 (2015)
Harris, Z.: Distributional structure. Word 10, 146–162 (1954)
Hedges, J., Sadrzadeh, M.: A generalised quantifier theory of natural language in categorical compositional distributional semantics with bialgebras. In: EPTCS Proceedings of the 13th International Conference on Quantum Physics and Logic, to appear (2016)
Herbelot, A., Ganesalingam, M.: Measuring semantic content in distributional vectors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 440–445. Association for Computational Linguistics (2013)
Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 655–665. Association for Computational Linguistics (2014)
Kartsaklis, D.: Compositional Distributional Semantics with Compact Closed Categories and Frobenius Algebras. Ph.D. thesis, University of Oxford (2015)
Kartsaklis, D.: Coordination in categorical compositional distributional semantics. In: EPTCS Proceedings of the Workshop on Semantic Spaces at the Intersection of NLP, Physics and Cognitive Science, to appear (2016)
Kartsaklis, D., Sadrzadeh, M.: Prior disambiguation of word tensors for constructing sentence vectors. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNL), pp. 1590–1601 (2013)
Kartsaklis, D., Sadrzadeh, M.: A Frobenius model of information structure in categorical compositional distributional semantics. In: Proceedings of the 14th Meeting on Mathematics of Language (2015)
Kartsaklis, D., Sadrzadeh, M., Pulman, S.: A unified sentence space for categorical distributional-compositional semantics: theory and experiments. In: COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Posters, 8–15 December 2012, Mumbai, India, pp. 549–558 (2012)
Kelly, G., Laplaza, M.: Coherence for compact closed categories. J. Pure Appl. Algebra 19(0), 193–213 (1980) http://www.sciencedirect.com/science/article/pii/0022404980901012
Kotlerman, L., Dagan, I., Szpektor, I., Zhitomirsky-Geffet, M.: Directional distributional similarity for lexical inference. Nat. Lang. Eng. 16(04), 359–389 (2010)
Lambek, J.: Type grammars as pregroups. Grammars 4(1), 21–39 (2001)
Lee, L.: Measures of distributional similarity. In: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, pp. 25–32 (1999)
MacCartney, B., Manning, C.D.: Natural logic for textual inference. In: ACL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics (2007)
MacLane, S.: Categories for the Working Mathematician. Springer, Berlin (1971)
Milajevs, D., Kartsaklis, D., Sadrzadeh, M., Purver, M.: Evaluating neural word representations in tensor-based compositional settings. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, pp. 708–719 (2014). http://www.aclweb.org/anthology/D14-1079
Mitchell, J., Lapata, M.: Composition in distributional models of semantics. Cogn. Sci. 34(8), 1388–1439 (2010)
Piedeleu, R.: Ambiguity in Categorical Models of Meaning. Master’s thesis, University of Oxford (2014)
Piedeleu, R., Kartsaklis, D., Coecke, B., Sadrzadeh, M.: Open system categorical quantum semantics in natural language processing. In: Proceedings of the 6th Conference on Algebra and Coalgebra in Computer Science. Nijmegen, Netherlands (2015)
Preller, A.: From Sentence to Concept, a Linguistic Quantum Logic. Tech. Rep. RR-11019. LIRMM (2011). http://hal-lirmm.ccsd.cnrs.fr/lirmm-00600428
Reddy, S., McCarthy, D., Manandhar, S.: An empirical study on compositionality in compound nouns. In: Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP-11) (2011)
Rimell, L.: Distributional lexical entailment by topic coherence. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26–30, 2014, Gothenburg, Sweden, pp. 511–519 (2014)
Rubenstein, H., Goodenough, J.: Contextual correlates of synonymy. Commun. ACM 8(10), 627–633 (1965)
Sadrzadeh, M., Clark, S., Coecke, B.: Frobenius anatomy of word meanings i: subject and object relative pronouns. J. Log. Comput. 23, 1293–1317 (2013)
Sadrzadeh, M., Clark, S., Coecke, B.: Frobenius anatomy of word meanings 2: possessive relative pronouns. J. Log. Comput. 26, 785–815 (2016)
Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Commun. ACM 18, 613–620 (1975)
Schütze, H.: Automatic word sense discrimination. Comput. Linguist. 24, 97–123 (1998)
Selinger, P.: Dagger compact closed categories and completely positive maps. Electronic Notes in Theoretical Computer Science 170, 139–163 (2007)
Socher, R., Huval, B., Manning, C., Ng, AY: Semantic compositionality through recursive matrix-vector spaces. In: Conference on Empirical Methods in Natural Language Processing 2012 (2012)
Turney, P.D.: Similarity of semantic relations. Comput. Linguist. 32(3), 379–416 (2006)
Weeds, J., Weir, D., McCarthy, D.: Characterising measures of lexical distributional similarity. In: Proceedings of the 20th international conference on Computational Linguistics. No. 1015, Association for Computational Linguistics (2004)
Widdows, D.: Geometry and meaning. Center for the Study of Language and Information/SRI (2004)
Acknowledgements
Sadrzadeh is supported by EPSRC CAF grant EP/J002607/1 and Kartsaklis by AFOSR grant FA9550-14-1-0079. Balkır was supported by a Queen Mary Vice Principal scholarship, when contributing to this project.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Sadrzadeh, M., Kartsaklis, D. & Balkır, E. Sentence entailment in compositional distributional semantics. Ann Math Artif Intell 82, 189–218 (2018). https://doi.org/10.1007/s10472-017-9570-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-017-9570-x
Keywords
- Distributional semantics
- Compositional distributional semantics
- Distributional inclusion hypothesis
- Density matrices
- Entailment
- Entropy