Abstract
This article explores the application of data-driven machine translation (MT) to sign languages (SLs). The provision of an SL MT system can facilitate communication between Deaf and hearing people by translating information into the native and preferred language of the individual. In this paper we address data-driven SL MT predominantly for Irish SL (ISL) but also for German SL (DGS/Deutsche Gebärdensprache). We take two different purpose-built corpora to feed our MaTrEx MT system and in a set of experiments translating both to and from the SLs, we investigate the effects of SL data on statistical MT (SMT). Exploiting the bidirectionality of the MaTrEx system, we demonstrate how additional modules, such as recognition and SL animation, can potentially build a full SL MT model for spoken and SL communication in addition to promising evaluation scores. A secondary focus of the article is on the two main issues affecting SL MT, those of transcription and evaluation. We offer a discussion on both these common problems before concluding.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Bauer B, Nießen S, Heinz H (1999) Towards an automatic sign language translation system. In: Proceedings of the international workshop on physicality and tangibility in interaction: towards new paradigms for interaction beyond the desktop, Siena, Italy
Bungeroth J, Stein D, Ney H, Morrissey S, Way A (2008) The ATIS Sign Language Corpus. In: Proceedings of the 3rd Workshop on the representation and processing of sign languages at the 6th international conference on Language Resources and Evaluation (LREC-08), Marrakech, Morocco
Cahill P, Carson-Berndsen J (2006) The Jess Blizzard Challenge 2006 Entry. In: Proceedings of the Blizzard Challenge 2006 Workshop, Interspeech 2006, Pittsburgh, PA
Chomsky N (2000) New horizons in the study of language and mind. Cambridge University Press, Cambridge
Conroy P (2006) Signing in & signing out: the education and employment experiences of deaf adults in Ireland. Research report, Irish Deaf Society, Dublin
Conway A, Veale T (1994) A linguistic approach to sign language synthesis. In: Cockton G, Draper SW, Weir GRS (eds) People and computers IX: Proceedings of the Human Computer Interface conference (HCI), Glasgow, Scotland, pp 211–222
Crasborn O, Zwitserlood I (2008) The Corpus NGT: an online corpus for professionals and laymen. In: Proceedings of the 3rd workshop on the representation and processing of sign languages at the 6th international conference on Language Resources and Evaluation (LREC-08), Marrakech, Morrocco, pp 44–49
Dandapat S, Morrissey S, Naskar SK, Somers H (2010) Statistically motivated example-based machine translation using translation memory. In: Proceedings of the 8th International Conference on Natural Language Processing (ICON 2010), Kharagpur, India
Dreuw P, Rybach D, Deselaers T, Zahedi M, Ney H (2007) Speech Recognition techniques for a sign language recognition system. In: Proceedings of Interspeech 2006, Antwerp, Belgium, pp 2513–2516
Dreuw P, Ney H, Martinez G, Crasborn O, Piater J, Miguel Moya J, Wheatley M (2010) The signspeak project—bridging the gap between signers and speakers. In: Proceedings of the international conference on language resources and evaluation, Valletta, Malta, pp 476–481
Fourie J (2006) The design of a generic signing avatar. Technical Report, University of Stellenbosch, South Africa
Grieve-Smith AB (1999) English to American sign language machine translation of weather reports. In: Proceedings of the Second High Desert Student Conference in Linguistics (HDSL2), Albuquerque, NM, pp 13–30
Hanke T (2002) ViSiCAST Deliverable D5-1: Interface Definitions. Manuscript, Hamburg, Germany
Hanke T (2004) HamNoSys—representing sign language data in language resources and language processing contexts. In: Workshop on the representation and processing of sign languages at the Languages and Resources Evaluation Conference (LREC 04), Lisbon, Portugal, pp 1–6
Hanke T, Storz J (2008) iLex—a database tool integrating sign language corpus linguistics and sign language lexicography. In: Proceedings of the 3rd Workshop on the representation and processing of sign languages at the 6th international conference on Language Resources and Evaluation (LREC-08), Marrakech, Morrocco
Hassan H, Ma Y, Way A (2007) MaTrEx: the DCU Machine Translation System for IWSLT 2007. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Trento, Italy, pp 69–75
Hemphill C, Godfrey J, Doddington G (1990) The ATIS Spoken Language Systems Pilot Corpus. In: Proceedings of the workshop on speech and natural language, Hidden Valley, PA, pp 96–101
Huenerfauth M (2004) Spatial and planning models of ASL classifier predicates for machine translation. In: Proceedings of the 10th international conference on Theoretical and Methodological Issues in Machine Translation (TMI-04), Baltimore, MD, pp 65–74
Huenerfauth M (2006) Generating American sign language classifier predicates for English-to-ASL machine translation. PhD thesis, University of Pennsylvania, Philadelphia, PA
Jones D, Gill P (1998) Breaking down language barriers. Br Med J 316(7127): 1280–1476
Kanthak S, Vilar D, Matusov E, Zens R, Ney H (2005) Novel reordering approaches in phrase-based statistical machine translation. In: Proceedings of the ACL workshop on building and using parallel texts at the 43rd annual meeting of the Association of Computational Linguistics (ACL-05), Ann Arbor, MI, pp 167–174
Kennaway J, Glauert J, Zwitserlood I (2007) Providing signed content on the Internet by synthesized animation. ACM Trans Comput-Hum Interact 14: 1–29
Kneser R, Ney H (1995) Improved backing-off for n-gram language modelling. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, vol 1, Detroit, MI, pp 181–184
Koehn P, Och F, Marcu D (2003) Statistical phrase-based translation. In: Proceedings of the combined human language technology conference series and the North American Chapter of the Association for Computational Linguistics Conference Series (HLT-NAACL), Edmonton, Canada, pp 48–54
Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R, Dyer C, Bojar O, Constantin A, Herbst E (2007) Moses: open source toolkit for statistical machine translation. In: Proceedings of Demonstration and Poster Sessions at the 45th annual meeting of the Association of Computational Linguistics (ACL-07), Prague, Czech Republic
Leeson L (2001) Aspects of verbal valency in Irish sign language. PhD thesis, University of Dublin, Trinity College, Dublin, Ireland
Leeson L (2003) Sign language interpreters: agents of social change in Ireland?. In: Cronin M, ÓCuilleanáin C (eds) Languages of Ireland. Four Courts Press, Dublin, pp 148–164
Leeson L, Saeed J, Macduff A, Byrne-Dunne D, Leonard C (2006) Moving heads and moving hands: developing a digital corpus of Irish sign language. In: Proceedings of information technology and telecommunications conference 2006, Carlow, Ireland
Leusch G, Ueffing N, Ney H (2006) CDER: efficient MT evaluation using block movements. In: Proceedings of the 11th European Chapter of the Association for Computational Linguistics (EACL-06), Trento, Italy, pp 241–248
Liddell S, Johnson RE (1989) American sign language: the phonological base. Sign Lang Stud 64: 195–277
Ma Y, Tinsley J, Hassan H, Du J, Way A (2008) MaTrEx: the DCU system for IWSLT 2008. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT 2008), Honolulu, HI, pp 26–33
Marshall I, Sáfár E (2002) Sign language generation using HPSG. In: Proceedings of the 9th international conference on Theoretical and Methodological Issues in Machine Translation (TMI-02), Keihanna, Japan, pp 105–114
Marshall I, Sáfár E (2003) A prototype text to British Sign Language (BSL) translation system. In: the 41st annual meeting of the Association of Computational Linguistics (ACL-03) conference, Sapporo, Japan, pp 113–116
Matusov E, Zens R, Vilar D, Mauser A, Popovic M, Ney H (2006) The RWTH machine translation system. In: Proceedings of the TC-STAR workshop on speech-to-speech translation, Barcelona, Spain, pp 31–36
Matusov E, Leusch G, Banchs RE, Bertoldi N, Dechelotte D, Federico M, Kolss M, Lee Y-S, Marino JB, Paulik M, Roukos S, Schwenk H, Ney H (2008) System combination for machine translation of spoken and written language. IEEE Trans Audio Speech Lang Process 16(7): 1222–1237
Morrissey S (2008a) Assistive technology for deaf people: translating into and animating Irish sign language. In: Proceedings of the 11th International Conference on Computers Helping People with Special Needs (ICCHP) Young Researchers’ Consortium, Linz, Austria
Morrissey S (2008b) Data-driven machine translation for sign languages. PhD thesis, Dublin City University, Dublin, Ireland
Morrissey S (2011) Assessing three representation methods for sign language machine translation and evaluation. In: Proceedings of the 15th annual meeting of the European Association for Machine Translation (EAMT 2011), Leuven, Belgium, pp 137–144
Morrissey S, Way A (2005) An example-based approach to translating sign language. In: Proceedings of the workshop in example-based machine translation (MT Summit X), Phuket, Thailand, pp 109–116
Morrissey S, Way A (2006) Lost in translation: the problems of using mainstream MT evaluation metrics for sign language translation. In: Proceedings of the 5th SALTMIL workshop on minority languages at the Languages and Resources Evaluation Conference (LREC 2006), Genoa, Italy, pp 91–98
Morrissey S, Way A (2007) Joining hands: developing a sign language machine translation system with and for the deaf community. In: Proceedings of the conference and workshop on assistive technologies for people with vision & hearing impairments (CVHI), Granada, Spain
Morrissey S, Way A, Cahill P, Carson-Berndsen J (2007a) A complete Irish sign language to speech translation system. In: IBM CASCON Dublin Symposium, Dublin, Ireland
Morrissey S, Way A, Stein D, Bungeroth J, Ney H (2007b) Combining data-driven MT systems for improved sign language translation. In: Proceedings of machine translation Summit XI, Copenhagen, Denmark, pp 329–336
Naqvi S (2007) End-user involvement in assistive technology design for the deaf–are artificial forms of sign language meeting the needs of the target audience?. In: Proceedings of the conference and workshop on assistive technologies for people with vision & hearing impairments (CVHI), Granada, Spain
Neidle C, Sclaroff S, Athitsos V (2001) SignStream TM: a tool for linguistic and computer vision research on visual–gestural language data. Behav Res Methods Instrum Comput 33(3): 311–320
Ó’Baoill D, Matthews PA (2000) The Irish Deaf Community (Volume 2): the structure of Irish sign language. The Linguistics Institute of Ireland, Dublin, Ireland
Och FJ, Ney H (2000) Improved statistical alignment models. In: The 38th annual meeting of the Association for Computation Linguistics (ACL-00), Hong Kong, China, pp 440–447
Och F, Ney H (2003) A systematic comparision of various statistical alignment models. Comput Linguist 29(1): 19–51
Papineni K, Roukos S, Ward T, Zhu W (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL-02), Philadelphia, PA, pp 311–318
Penkale S, Haque R, Dandapat S, Banerjee P, Srivastava AK, Du J, Pecina P, Naskar SK, Forcada ML, Way A (2010) MATREX: The DCU MT System for WMT 2010. In: Proceedings of the joint fifth workshop on statistical machine Translation and MetricsMATR, Uppsala, Sweden, pp 149–154
Pierce J, Carroll J, Hamp E, Hays D, Hockett C, Oettinger A, Perlis A (1966) Language and machines: computers in translation and linguistics. Technical Report: Automatic Language Processing Committee, National Academy of Sciences, National Research Council. Washington, DC
Pinker S (1994) The language instinct: how the mind creates language. Penguin, London
Pizzuto E, Rossini P, Russo T (2006) Representing signed languages in written form: questions that need to be posed. In: Proceedings of the 2nd workshop on the representation and processing of sign languages at the Languages and Resources Evaluation Conference (LREC 2006), Genoa, Italy, pp 0–13
Prillwitz S, Leven R, Zienert H, Hanke T, Henning J (1989) HamNoSys, Version 2.0. Hamburg notation system for sign languages: an introductory guide. Hamburg Signum, Hamburg
Rabiner LR, Juang BH (1989) An introduction to hidden Markov models. IEEE ASSP Mag 4(1): 4–16
Sáfár E, Marshall I (2002) The architecture of an English-Text-to-Sign-languages translation system. In: Proceedings of the international conference on Recent Advances in Natural Language Processing (RANLP-02), Tzigov Chark, Bulgaria, pp 223–228
San-Segundo R, Montero JM, Macias-Guarasa J, Córdoba R, Ferreiros J, Pardo JM (2007) Proposing a speech to gesture translation architecture for Spanish deaf people. J Vis Lang Comput 19:523–538
San-Segundo R, López V, Martín R, Sánchez D, García A (2010) Language resources for Spanish—Spanish sign language (lse) translation. In: Proceedings of the 4th workshop on the representation and processing of sign languages: corpora and sign language technologies at LREC 2010, Valetta, Malta, pp 208–211
Shneiderman B (1998) Designing user interfaces. Addison Wesley, Menlo Park, CA
Somers H (2006) Language engineering and the pathway to healthcare: a user-oriented view. In: HLT/NAACL-06 medical speech translation, Proceedings of the Workshop, New York, NY, pp 32–39
Somers H, Lovel H (2003) Computer-based support for patients with limited English. In: Association for computational linguistics EACL 2003, 10th Conference of the European Chapter, Proceedings of the 7th International EAMT Workshop on MT and other language technology tools: improving MT through other language tools, resources and tools for building MT, Budapest, Hungary, pp 41–49
Speers D (2001) Representation of American sign language for machine translation. PhD thesis, Georgetown University, Washington, DC
Stein D, Bungeroth J, Ney H (2006) The architecture of an English text-to-sign languages translation system. In: Proceedings of the 11th annual conference of the European Association for Machine Translation (EAMT, ’06), Oslo, Norway, pp 169–177
Stein D, Dreuw P, Ney H, Morrissey S, Way A (2007) Hand in hand: automatic sign language to English translation. In: Proceedings of the 11th international conference on Theoretical and Methodological Issues in Machine Translation (TMI-07), Skövde, Sweden, pp 214–220
Stein D, Schmidt C, Ney H (2010) Sign language machine translation overkill. In: Federico M, Lane I, Paul M, Yvon F (eds) International workshop on spoken language translation, Paris, France, pp 337–344
Stein D, Schmidt C, Ney H (2012) Analysis, preparation and optimization of statistical sign language machine translation. Mach Translat 26:325–357
Stokoe WC (1960) Sign language structure: an outline of the visual communication system of the American deaf. Studies in Linguistics, Occasional Paper, 2nd printing 1993. Linstok Press, Burtonsville, MD
Stokoe WC (1972) Semiotics and human sign languages. Mouton & Co. N.V. Publishers, The Hague
Stroppa N, Way A (2006) MaTrEx: DCU Machine translation system for IWSLT 2006. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Kyoto, Japan, pp 31–36
Sutton V (1995) Lessons in sign writing, textbook and workbook (2nd edn). The Center for Sutton Movement Writing Inc., La Jolla, CA
Tinsley J, Ma Y, Ozdowska S, Way A (2008) MaTrEx: the DCU MT system for WMT 2008. In: Proceedings of the third workshop on statistical machine translation, ACL 2008, Colombus, OH, pp 171–174
Traxler C (2000) The Stanford Achievement Test, 9th Edition: national norming and performance standards for deaf and hard-of-hearing students. J Deaf Stud Deaf Educ 5(4): 337–348
van Slype G (1979) Critical methods for evaluating the quality of machine translation. Technical Report: BR-19142, European Commission Directorate General Scientific and Technical Information and Information Management, Bureau Marcel van Dijk, Brussells, Belgium
van Zijl L, Combrink A (2006) The South African sign language machine translation project: issues on non-manual sign generation. In: Proceedings of South African Institute for Computer Scientists and Information Technologists Conference (SAICSIT 06), Cape Winelands, South Africa, pp 127–134
van Zijl L, Olivrin G (2008) South African sign language assistive translation. In: Proceedings of IASTED international conference on assistive technologies, Baltimore, MD
Veale T, Conway A, Collins B (1998) The challenges of cross-modal translation: English to sign language translation in the Zardoz system. Mach Transl 13(1): 81–106
Vilar D, Stein D, Huck M, Ney H (2010) Jane: open source hierarchical translation, extended with reordering and Lexicon models. In: ACL 2010 joint fifth workshop on statistical Machine Translation and Metrics (MATR), Uppsala, Sweden, pp 262–270
Ward W (1991) Understanding spontaneous speech: the phoenix system. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, Toronto, Canada, pp 365–367
Weaver W (1949) Recent contributions to the mathematical theory of communication. In: The mathematical theory of communication. The University of Illinois Press, Urbana IL, pp 94–117
Wu C-H, Su H-Y, Chiu Y-H, Lin C-H (2007) Transfer-based statistical translation of Taiwanese sign language using PCFG. In: ACM Transactions on Asian Language Information Processing (TALIP), vol 6, No. 1
Zens R, Ney H (2008) Improvements in dynamic programming beam search for phrase-based statistical machine translation. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT 08), Honolulu, Hawaii, pp 195–205
Zhao L, Kipper K, Schuler W, Vogler C, Badler N, Palmer M (2000) A machine translation system from English to American sign language. In: Envisioning machine translation in the information future: proceedings of the fourth conference of the Association for Machine Translation (AMTA-00), Cuernavaca, Mexico, pp 293–300
Author information
Authors and Affiliations
Corresponding author
Additional information
The work described in this article was, for the most part, carried out while at Dublin City University.
Rights and permissions
About this article
Cite this article
Morrissey, S., Way, A. Manual labour: tackling machine translation for sign languages. Machine Translation 27, 25–64 (2013). https://doi.org/10.1007/s10590-012-9133-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10590-012-9133-1