Abstract
The aim of this research is to generate measurable evaluation criteria acceptable to chatbot users. Results of two studies are summarised. In the first, fourteen participants were asked to do a critical incident analysis of their transcriptions with an ELIZA-type chatbot. Results were content analysed, and yielded seven overall themes. In the second, these themes were made into statements of an attitude-like nature, and 20 participants chatted with five winning entrants in the 2011 Chatterbox Challenge and five which failed to place. Latent variable analysis reduced the themes to four, resulting in four subscales with strong reliability which discriminated well between the two categories of chatbots. Content analysis of freeform comments led to a proposal of four dimensions along which people judge the naturalness of a conversation with chatbots.
Chapter PDF
Similar content being viewed by others
Keywords
References
Aron, J.: How innovative is Apple’s new voice assistant, Siri? New Scientist 212 (2386), 24 (2011)
Barwise, J., Perry, J.: Situations and Attitudes. Mass: MIT Press, Cambridge (1983)
Binet, Etude Expérimentale de l’Intelligence. Schleicher Frères & Cie, Paris (1903)
Brooks, R.: Intelligence without representation. Artificial Intelligence 47, 139–159 (1991)
Charette, R.: WellPoint Hires IBM’s “Dr.” Watson (April 11, 2013), spectrum.ieee.org/riskfactor/biomedical/diagnostics/wellpoint-hires-ibms-dr-watson
Charmaz, K.: Constructing Grounded Theory. Sage, London (2006)
Cohen, P.: If Not Turing’s Test, Then What? AI Magazine 26(4), 61–67 (2005)
Flanagan, J.: The critical incident technique. Psychological Bulletin 51(2), 327–358 (1954)
Fodor, J.: The Language of Thought. Thomas Crowell, London (1975)
French, R.M.: Subcognition and the limits of the Turing Test. Mind 99, 53–65 (1990)
Hodges, A.: Alan Turing: the Enigma. Burnett, London (1983)
Hung, V., Elvir, M., Gonzalez, A.J., DeMara, R.F.: A Method For Evaluating Naturalness in Conversational Dialog Systems. In: Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics, pp. 2647–2652 (2009)
ISO/IEC, Ergonomic Requirements for Office Work with Visual Display Terminals (VDT)s - Part 11 Guidance on Usability. ISO-IEC, Geneva (1998)
Kirakowski, J., O’Donnell, P., Yiu, A.: Establishing the hallmarks of a convincing chatbot-human dialogue. In: Maurtua, I. (ed.) Human- Computer Interaction, In-Teh., Vukovar (2009)
Rzepka, R., Ge, Y., Araki, K.: Naturalness of an utterance based on the automatically retrieved commonsense. In: Proceedings of the Nineteenth IJCAI (2005)
Semeraro, G., Andersen, H.H.K., Andersen, V., Lops, P., Abbattista, F.: Evaluation and validation of a conversational agent embodied in a bookstore. In: Carbonell, N., Stephanidis, C. (eds.) UI4ALL 2002. LNCS, vol. 2615, pp. 360–371. Springer, Heidelberg (2003)
Shawar, B.A., Atwell, E.: Chatbots: Are they Really Useful? LDV Forum 22 (1), 29–49 (2007)
Shieber, S.M.: Does the Turing Test Demonstrate Intelligence or Not? Proceedings of the 21st National Conference on Artificial Intelligence 2, 1539–1542 (2006)
Turing, A.: Computing machinery and intelligence. Mind 59, 433–460 (1950)
Weizenbaum, J.: ELIZA – A Computer Program for the Study of Natural Language Communication Between Man And Machine. Communications of the ACM 9(1), 36–45 (1966)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Morrissey, K., Kirakowski, J. (2013). ‘Realness’ in Chatbots: Establishing Quantifiable Criteria. In: Kurosu, M. (eds) Human-Computer Interaction. Interaction Modalities and Techniques. HCI 2013. Lecture Notes in Computer Science, vol 8007. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39330-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-39330-3_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39329-7
Online ISBN: 978-3-642-39330-3
eBook Packages: Computer ScienceComputer Science (R0)