Measuring the User Experience of Computer-Aided Translation Systems: A Comparative Study

Abstract

Computer-aided translation (CAT) has become the mainstream mode of translation in the language industry. In the process of translation, the CAT systems’ irritating features often cause considerable cognitive friction on the part of translators. In this context, this study evaluates and compares, qualitatively and quantitatively, the user experience of four CAT systems, namely, SDL Trados Studio, memoQ, Snowman CAT, and YiCAT. Over 100 graduate students in translation from a Chinese university completed a battery of tasks. They then filled out a questionnaire, which included the standardised System Usability Scale, preferences for the four CAT systems based on key functions, and open-ended questions regarding each CAT system's weaknesses and strengths. The study seeks to raise CAT vendors’ awareness of UX and find out what usability problems translators have encountered. It has implications for CAT tool development, pedagogy as well as cognitive translation studies.

Appendices

Appendix 1: The System Usability Scale (SUS)

(Notes: 1. A Chinese translation of SUS by this author was provided to facilitate the students’ understanding; 2. See Fig. 1 for the specific 5-point Likert scale format adopted in this survey.)

• 1. I think that I would like to use this system frequently [我愿意经常使用这款软件].

• 2. I found the system unnecessarily complex [我觉得该软件过于复杂].

• 3. I thought the system was easy to use [我认为该软件易于使用].

• 4. I think that I would need the support of a technical person to be able to use this system [我想我需要技术人员的支持才能使用该软件].

• 5. I found the various functions in this system were well integrated [我发现该软件的各种功能都很好地集成在一起].

• 6. I thought there was too much inconsistency in this system [我认为该软件在程序设计、界面外观、帮助文档等方面有太多前后矛盾之处].

• 7. I would imagine that most people would learn to use this system very quickly [我想大多数人会很快学会使用该软件].

• 8. I found the system very cumbersome to use [我发现该软件使用起来非常麻烦].

• 9. I felt very confident using the system [我对使用该软件感到非常有信心].

• 10. I needed to learn a lot of things before I could get going with this system [在使用该软件之前, 我需要学习很多东西].

Appendix 2: Test Tasks to Be Completed Using Each of the Four CAT Systems

(Note: The task description was originally in Chinese. This is a generic version; in the actual test, one customised task description with cheat-sheet reminders was provided for each CAT system.)

• 1. Create a new project, choose SourceText_English.docx, and create a new translation memory (TM) and termbase.

• 2. Align alignment_Chinese.docx and alignment_English.docx, and export the aligned pairs into the TM.

• 3. Pre-translate the source text (ST).

• 4. Translate the ST sentence by sentence in the Editor view by copying the translation from the provided reference target text, confirm the segments, add the three terms—"Lao Dao”, “Peng Li”, and “chow mein"—to the termbase, then perform a concordance search for “jujubes” and add its translation to the termbase.

• 5. Review the translation, perform quality assurance, and make sure the translation progress is 100%.

• 6. Open the TM and replace all the instances of the translation (“老刀”) of “Lao Dao” with “老道”.

• 7. Export the target text and the translation memory (in .tmx format).