Abstract
Words not present in the dictionary are almost always found in unrestricted texts. However, there is a need to obtain their likely base forms (in lemmatization), or morphological categories (in tagging), or both. Some of them find their ways into dictionaries, and it would be nice to predict what their entries should look like. Humans can perform those tasks using endings of words (sometimes prefixes and infixes as well), and so can do computers. Previous approaches used manually constructed lists of endings and associated information. Brill proposed transformation-based learning from corpora, and Mikheev used Brill’s approach on data for a morphological lexicon. However, both Brill’s algorithm, and Mikheev’s algorithm that is derived from Brill’s one, lack speed, both in the rule acquisition phase, and in the rule application phase. Their algorithms handle only the case of tagging, although an extension to other tasks seems possible. We propose a very fast finite-state method that handles all of the tasks described above, and that achieves similar quality of guessing.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Eric Brill. A Corpus-Based Approach to Language Learning. PhD thesis, Department of Computer and Information Science, University of Pennsylvania, USA, 1993.
Eric Brill. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565, December 1995.
Andrei Mikheev. Automatic rule induction for unknown-word guessing. Computational Linguistics, 23(3):405–423, September 1997.
Dominique Petitpierre and Graham Russell. MMORPH-the Multext morphology program. Technical report, ISSCO, 54 route des Acacias, CH-1227 Carouge, Switzerland, October 1995.
Emmanuel Roche and Yves Schabes. Deterministic part-of-speech tagging with finite-state transducers. Computational Linguistics, 21(2):227–253, June 1995.
Jan Tokarski. Schematyczny indeks a tergo polskich form wyrazowych. Wydawnictwo Naukowe PWN, 1993.
Ralph Weischedel, Marie Meteer, Richard Schwartz, Lance Ramshaw, and Jeff Palmucci. Coping with ambiguity and unknown words through probabilistic models. Computational Linguistics, 19(2):359–382, 1993.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Daciuk, J. (2001). Treatment of Unknown Words. In: Boldt, O., Jürgensen, H. (eds) Automata Implementation. WIA 1999. Lecture Notes in Computer Science, vol 2214. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45526-4_7
Download citation
DOI: https://doi.org/10.1007/3-540-45526-4_7
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42812-1
Online ISBN: 978-3-540-45526-4
eBook Packages: Springer Book Archive