Abstract
The evaluation of interactive adaptive systems has long been acknowledged to be a complicated and demanding endeavour. Some promising approaches in the recent past have attempted tackling the problem of evaluating adaptivity by “decomposing” and evaluating it in a “piece-wise” manner. Separating the evaluation of different aspects can help to identify problems in the adaptation process. This paper presents a framework that can be used to guide the “layered” evaluation of adaptive systems, and a set of formative methods that have been tailored or specially developed for the evaluation of adaptivity. The proposed framework unifies previous approaches in the literature and has already been used, in various guises, in recent research work. The presented methods are related to the layers in the framework and the stages in the development lifecycle of interactive systems. The paper also discusses practical issues surrounding the employment of the above, and provides a brief overview of complementary and alternative approaches in the literature.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Ackerman, M.S., Cranor, L.F., Reagle, J.: Privacy in e-commerce: examining user scenarios and privacy preferences. In: 1st ACM Conference on Electronic Commerce, pp. 1–8. ACM, Denver, CO (1999)
Albrecht D., Zuckerman I.: Statistical and probabilistic methods for user modeling. Special issue User Model. User-Adap. Inter. 17(1–2), 1–215 (2007)
Anderson, J.R., Boyle, C.F., Yost, G.: The geometry tutor. In: 9th International Joint Conference on Artificial Intelligence, Los Angeles, CA, pp. 1–7. Morgan Kaufmann, San Francisco (1985)
Arruabarrena, R., Pérez, T., López-Cu adrado, J., Gutiérrez, J., Vadillo, J.: On evaluating adaptive systems for education. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Malaga, Spain. LNCS, vol. 2347, pp. 363–367. Springer, Berlin (2002)
Batliner A., Steidl S., Hacker C., Nöth E.: Private emotions versus social interaction: a data-driven approach towards analysing emotion in speech. User Model. User-Adap. Inter. 18(1-2), 175–206 (2008)
Berkovsky S., Kuflik T., Ricci F.: Mediation of user models for enhanced personalization in recommender systems. User Model. User-Adap. Inter. 18(3), 245–286 (2008)
Beyer H., Holtzblatt K.: Contextual Design: Defining Customer-Centred Systems. Morgan Kaufmann, San Francisco (1997)
Billsus, D., Pazzani, M.: A hybrid user model for news story classification. In: 7th International Conference on User Modeling, Banff, Canada, pp. 98–108. Springer, Vienna (1999)
Bontcheva K., Wilks Y.: Tailoring automatically generated hypertext. User Model. User-Adap. Inter. 15(1-2), 135–168 (2005)
Boyle C., Encarnacion A.O.: MetaDoc: an adaptive hypertext reading system. User Model. User-Adap. Inter. 4(1), 1–19 (1994)
Brown E.J., Brailsford T.J., Fisher T., Moore A.: Evaluating learning style personalization in adaptive systems: quantitative methods and approaches. IEEE Trans. Learn. Technol. 2(1), 10–22 (2009)
Browne D., Norman M., Riches D.: Why build adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 15–57. Academic Press, London (1990)
Brusilovsky P., Eklund J.: A study of user model based link annotation in educational hypermedia. J. Univ. Comput. Sci. 4(4), 429–448 (1998)
Brusilovsky, P., Karagiannidis, C., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 1–8. Sonthofen, Germany (2001)
Brusilovsky, P., Farzan, R., Ahn, J.: Layered evaluation of adaptive search. In: Workshop on Evaluating Exploratory Search Systems at SIGIR06, pp. 11–13. Seattle, WA (2006)
Bull S., Kay J.: Student models that invite the learner in: the SMILI Open learner modelling framework. Int. J. Artif. Intell. Edu. 17(2), 89–120 (2007)
Card S.K., Thomas T.P., Newell A.: The Psychology of Human–Computer Interaction. Lawrence Erbaum Associates, London (1983)
Carmagnola F., Cena F., Console L., Cortassa O., Gena C., Goy A., Torre I., Toso A., Vernero F.: Tag-based user modeling for social multi-device adaptive guides. User Model. User-Adap. Inter. 18(5), 497–538 (2008)
Carmichael D.J., Kay J., Kummerfeld B.: Consistent modelling of users, devices and sensors in a ubiquitous computing environment. User Model. User-Adap. Inter. 15(3-4), 197–234 (2005)
Carroll M.J.: Five reasons for scenario-based design. Interact. Comput. 13(1), 43–60 (2000)
Cena F., Console L., Gena C., Goy A., Levi G., Modeo S., Torre I.: Integrating heterogeneous adaptation techniques to build a flexible and usable mobile tourist guide. AI Commun. 19(4), 369–384 (2006)
Chen H.T.: A comprehensive typology for program evaluation. Am. J. Eval. 17(2), 121–130 (1996)
Cheverst K., Byun H. E., Fitton D., Sas C., Kray C., Villar N.: Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Model. User-Adap. Inter. 15(3-4), 235–273 (2005)
Chickering D.M., Paek T.: Personalizing influence diagrams: applying online learning strategies to dialogue management. User Model. User-Adap. Inter. 17(1-2), 71–91 (2007)
Chin D.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adap. Inter. 11(1-2), 181–194 (2001)
Claypool, M., Le, P., Wased, M., Brown, D.: Implicit interest indicators. In: International Conference on Intelligent User Interfaces, pp. 33–40. ACM, Santa Fe, NM (2001)
Conati C., MacLaren H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adap. Inter. 19(3), 267–303 (2009)
Cooper A.: The Inmates are Running the Asylum. Macmillan, Indianapolis (1999)
Cooper, D., Arroyo, I., Woolf, B., Muldner, K., Burleson, W., Christopherson, R.: Sensors model student self concept in the classroom. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 30–41. Springer, Berlin (2009)
Cramer H., Evers V., Ramlal S., van Someren M., Rutledge L., Stash N., Aroyo L., Wielinga B.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adap. Inter. 18(5), 455–496 (2008)
Cronbach L.J., Ambron S.R., Dornbusch S.M., Hess R.D., Hornik R.C., Philips D.C., Walker D.F., Weiner S.S.: Toward Reform of Program Evaluation. Jossey-Bass, San Francisco (1980)
Czarkowski, M.: A scrutable adaptive hypertext. PhD Thesis, University of Sydney (2006)
Damiano R., Gena C., Lombardo V., Nunnari F., Pizzo A.: A stroll with Carletto: adaptation in drama-based tours with virtual characters. User Model. User-Adap. Inter. 18(5), 417–453 (2008)
De Bra, P., Houben, G., Wu, H.: AHAM: a Dexter-based reference model for adaptive hypermedia. In: 10th ACM Conference on Hypertext and Hypermedia, pp. 147–156. ACM, Darmstadt, Germany (1999)
de Campos L., Fernández-Luna J., Huete J., Rueda-Morales M.: Managing uncertainty in group recommending processes. User Model. User-Adap. Inter. 19(3), 207–242 (2009)
de Rosis, F., Mazzotta, I., Miceli, M., Poggi, I.: Persuasion artifices to promote wellbeing. In: 1st International Conference on Persuasive Technology, Eindhoven, The Netherlands. LNCS, vol. 3962, pp. 84–95. Springer, Berlin (2006)
Degemmis M., Lops P., Semeraro G.: A content-collaborative recommender that exploits WordNet-based user profiles for neighborhood formation. User Model. User-Adap. Inter. 17(3), 217–255 (2007)
Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: Workshop on the What, Who, Where, When, and How of Context-Awareness at CHI 2000, pp. 304–307. The Hague, The Netherlands (2000)
Dix A., Finlay J., Abowd G., Beale R.: Human Computer Interaction. 2nd edn. Prentice-Hall, Englewood Cliffs (1998)
D’Mello S.K., Craig S.D., Sullins J., Graesser A.C.: Predicting affective states through an emote-aloud procedure from AutoTutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Edu. 16(1), 3–28 (2006)
D’Mello S.K., Craig S.D., Witherspoon A., McDaniel B., Graesser A.C.: Automatic detection of learner’s affect from conversational cues. User Model. User-Adap. Inter. 18(1-2), 45–80 (2008)
Domshlak C., Joachims T.: Efficient and non-parametric reasoning over user preferences. User Model. User-Adap. Inter. 17(1-2), 41–69 (2007)
Dumas J.S., Loring B.A.: Moderating Usability Tests: Principles and Practices for Interacting. Morgan Kaufmann, San Francisco (2008)
Encarnação, L., Stoev, S.: Application-independent intelligent user support system exploiting action-sequence based user modeling. In: 7th International Conference on User Modeling, pp. 245–254. Banff, Canada (1999)
Ericsson, K.A., Simon, H.A.: Protocol analysis: verbal reports as data, revised edition. MIT Press, Cambridge, MA (1993)
Forbes-Riley K., Rotaru M., Litman D.J.: The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Model. User-Adap. Inter. 18(1-2), 11–43 (2008)
Gena C.: Methods and techniques for the evaluation of user-adaptive systems. Knowl. Eng. Rev. 20(1), 1–37 (2005)
Gena, C., Ardissono, L.: Intelligent support to the retrieval of information about hydric resources. In: 3rd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems. LNCS, vol. 3137, pp. 126–135. Springer, Berlin (2004)
Gena C., Weibelzahl S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 720–762. Springer, Berlin (2007)
George S., Zukerman I., Niemann M.: Inferences, suppositions and explanatory extensions in argument interpretation. User Model. User-Adap. Inter. 17(5), 439–474 (2007)
Glahn, C., Specht, M., Koper, R.: Smart indicators on learning interactions. In: 2nd European Conference on Technology Enhanced Learning, Crete, Greece. LNCS, vol. 4753, pp. 56–70. Springer, Berlin (2007)
Goecks, J., Shavlik, J.: Learning users’ interests by unobtrusively observing their normal behavior. In: 5th International Conference on Intelligent User Interfaces, pp. 129–132. ACM, New Orleans, LA (2000)
Goren-Bar D., Graziola I., Pianesi F., Zancanaro M.: The influence of personality factors on visitor attitudes towards adaptivity dimensions for mobile museum guides. User Model. User-Adap. Inter. 16(1), 31–62 (2006)
Goren-Bar, D., Graziola, I., Rocchi, C., Pianesi, F., Stock, O., Zancanaro, M.: Designing and redesigning an affective interface for an adaptive museum guide. In: 1st International Conference on Affective Computing and Intelligent Interaction, Beijing, China. LNCS, vol. 3784, pp. 939–946. Springer, Berlin (2005)
Gould J.D., Lewis C.: Designing for usability: key principles and what designers think. Commun. ACM 28(3), 300–311 (1985)
Gould, J., Conti, J., Hovanyecz, T.: Composing letters with a simulated listening typewriter. In: 1st ACM Conference on Human Factors in Computer Systems (CHI), pp. 367–370. ACM, Gaithersburg, MD (1982)
Green P.D., Ha E.S., Bullock G.J.: Enough already about “black box” experiments: Studying mediation is more difficult than most scholars suppose. Ann. Am. Acad. Political Soc. Sci. 628(1), 200–208 (2010)
Grudin, J., Pruitt, J.: Personas, participatory design, and product development: An infrastructure for engagement. In: Participatory Design Conference, pp. 144–161, ACM, Malmö, Sweden (2002)
Guzmán E., Conejo R., Pérez de la Cruz J.-L.: Adaptive testing for hierarchical student models. User Model. User-Adap. Inter. 17(1-2), 119–157 (2007)
Herder, E.: Utility-based evaluation of adaptive systems. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 25–30. Johnstown, PA, USA (2003)
Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)
Hertzum M., Hansen K., Andersen H.H.K.: Scrutinizing usability evaluation: does thinking aloud affect behaviour and mental workload?. Behav. Inf. Technol. 28(2), 165–181 (2009)
Hollink V., van Someren M., Wielinga B.: Discovering stages in web navigation for problem-oriented navigation support. User Model. User-Adap. Inter. 17(1-2), 183–214 (2007)
Höök K.: Steps to take before intelligent user interfaces become real. Interact. Comput. 12(4), 409–426 (2000)
Hoppe, H., Tauber, M., Ziegler, J.: A survey of models and formal description methods in HCI with example applications. In: ESPRIT Project, vol. 385 (1986)
Horvitz E., Paek T.: Complementary computing: Policies for transferring callers from dialog systems to human receptionists. User Model. User-Adap. Inter. 17(1-2), 159–182 (2007)
Jameson A.: Systems that Adapt to their Users: An Integrative Perspective. Saarland University, Saarbrücken (2001)
Jameson A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 305–330. L. Erlbaum Associates, Hillsdale (2003)
Jameson, A.: User modeling meets usability goals. In: 10th International Conference on User Modeling, Edinburgh, UK. LNAI, vol. 3538, pp. 1–3. Springer, Berlin (2005)
Jameson A.: Adaptive user interfaces and agents. 2nd edn. In: Sears, A., Jacko, J. (eds) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 433–458. CRC Press, Boca Raton (2008)
Jameson A.: Understanding and dealing with usability side effects of intelligent processing. AI Mag. 30(4), 23–40 (2009)
Jameson A., Schwarzkopf, E.: Pros and cons of controllability: An empirical study. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 193-202. Springer, Berlin (2002)
Kaplan C., Fenwick J., Chen J.: Adaptive hypertext navigation based on user goals and context. User Model. User-Adap. Inter. 3(3), 193–220 (1993)
Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive applications and services. In: 1st International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Trento, Italy. LNCS, vol. 1892, pp. 343–346. Springer, Berlin, (2000)
Kay, J.: Stereotypes, student models and scrutability. 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1839, pp. 19–30. Springer, Berlin, (2000)
Kay J.: Learner control. User Model. User-Adap. Inter. 11(1-2), 111–127 (2001)
Knutov E., De Bra P., Pechenizkiy M.: AH—12 years later: a comprehensive survey of adaptive hypermedia methods and techniques. New Rev. Hyperme’d. Multime’d. 15(1), 5–38 (2009)
Kobsa A.: Privacy-enhanced web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 628–670. Springer, Berlin (2007)
Koch, N., Wirsing, M.: The Munich reference model for adaptive hypermedia applications. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 213–222. Springer, Berlin (2002)
Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: 14th International Joint Conference on Artificial Intelligence, Montréal, Canada, pp. 1137–1145. Morgan Kaufmann, San Francisco (1995)
Kosba E., Dimitrova V., Boyle R.: Adaptive feedback generation to support teachers in web-based distance education. User Model. User-Adap. Inter. 17(4), 379–413 (2007)
Krogsæter M., Oppermann R., Thomas G.C.: A user interface integrating adaptability and adaptivity. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 97–125. Lawrence Erlbaum, Hillsdale, NJ (1994)
Krueger R., Casey M.: Focus Groups: A Practical Guide for Applied Research. 4th edn. Sage Publications, Los Angeles (2009)
Kruppa, M., Aslan, I.: Parallel presentations for heterogeneous user groups—an initial user study. In: 4th International Conference on Intelligent Technologies for Interactive Entertainment, Madonna di Campiglio, Italy. LNAI. vol. 3814, pp. 54–63, Springer, Berlin (2005)
Law A., Freer Y., Hunter J., Logie R., McIntosh N., Quinn J.: A comparison of graphical and textual presentations of time series data to support medical decision making in the neonatal intensive care unit. J. Clin. Monitor. Comput. 19(3), 183–194 (2005)
Lekakos G., Giaglis G.: A hybrid approach for improving predictive accuracy of collaborative filtering algorithms. User Model. User-Adap. Inter. 17(1-2), 5–40 (2007)
Lewis C.: Using the ‘thinking-aloud’ method in cognitive interface design. In: Research Report RC9265. IBM T.J. Watson Research Center, Yorktown Heights, NY (1982)
Ley, T., Kump, B., Maas, A., Maiden, N., Albert, D.: Evaluating the adaptation of a learning system before the prototype is ready: A paper-based lab study. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 331–336, Springer, Berlin (2009)
Limongelli, C., Sciarrone, F., Vaste, G.: LS-Plan: An effective combination of dynamic courseware generation and learning styles in web-based education. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 133–142, Springer, Berlin (2008)
MacLaren, B., Koedinger, K.: When and why does mastery learning work: instructional experiments with ACT-R “SimStudents”. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France, pp. 355–366. Springer, Berlin (2002)
Magoulas, G.D., Chen, S.Y., Papanikolaou, K.A.: Integrating layered and heuristic evaluation for adaptive learning environments. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 5–14, Johnstown, PA (2003)
Maguire M.: Methods to support human-centred design. Int. J. Hum. Comput. Stud. 55(4), 587–634 (2001)
Masthoff J.: The evaluation of adaptive systems. In: Patel, N. (ed.) Adaptive Evolutionary Information Systems, pp. 329–347. Idea Group Publishing, London (2002)
Masthoff J.: Group modeling: slecting a sequence of television items to suit a group of viewers. User Model. User-Adap. Inter. 14(1), 37–85 (2004)
Masthoff, J.: The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. In: 5th Workshop on User-Centred Design and Evaluation of Adaptive Systems at AH06, pp. 460–469. Dublin, Ireland (2006)
Masthoff, J.: Automatically constructing good hierarchies: HCI meets AI, unpublished
Masthoff J., Gatt A.: In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems. User Model. User-Adap. Inter. 16(3-4), 281–319 (2006)
Masthoff, J., Vasconcelos, W.W., Aitken, C., Correa da Silva, F.S.: Agent-based group modelling for ambient intelligence. In: AISB Symposium on Affective Smart Environments, Newcastle, UK (2007)
Maulsby, D., Greenberg, S., Mander, R.: Prototyping an intelligent agent through wizard of Oz. In: 10th ACM Conference on Human Factors in Computing Systems, pp. 277–284. ACM, Amsterdam, The Netherlands (1993)
McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: CHI Work in Progress, Montréal, Canada (2006)
Miettinen M., Oulasvirta A.: Predicting time-sharing in mobile interaction. User Model. User-Adap. Inter. 17(5), 475–510 (2007)
Millán E., Pérez de la Cruz J.L.: Diagnosis algorithm for student modeling diagnosis and its evaluation. User Model. User-Adap. Inter. 12(2–3), 281–330 (2002)
Mobasher B.: Data mining for web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 90–135. Springer, Berlin (2007)
: Special issue on data mining for personalization. User Model. User-Adap. Inter. 19(1-2), 1–166 (2009)
Moncur, W., Masthoff, J., Reiter, E.: What do you want to know? Investigating the information requirements of patient supporters. In: 21th IEEE International Symposium on Computer-Based Medical Systems, pp. 443–448. IEEE, Jyväskylä, Finland (2008)
Murray T.: Formative qualitative evaluation for ‘exploratory’ ITS research. Int. J. Artif. Intell. Educ. 4(2-3), 179–207 (1993)
Nguyen, H., Santos, E. Jr.: An evaluation of the accuracy of capturing user intent for information retrieval. In: International Conference on Artificial Intelligence, pp. 341–350. CSREA Press, Las Vegas, NV (2007)
Nguyen, H., Masthoff, J., Edwards, P.: Modelling a receiver’s position to persuasive arguments. In: 2nd International Conference on Persuasive Technology, Palo Alto, CA. LNCS, vol. 4744, pp. 271–282. Springer, Berlin (2007)
Nielsen J.: Evaluating the thinking-aloud technique for use by computer scientists. In: Hartson H.R., Hix D. (eds.) Advances in Human–Computer interaction, vol.3, pp. 69–82. Ablex, Norwood, NJ (1993)
Nielsen J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 25–64. John Wiley & Sons, New York (1994a)
Nielsen J.: Usability Engineering. 2nd edn. Morgan Kaufmann, San Francisco (1994b)
Norman A.D.: How might people interact with agents. Commun. ACM 37(7), 68–71 (1994)
Nückles M., Winter A., Wittwer J., Herbert M., Hübner S.: How do experts adapt their explanations to a layperson’s knowledge in asynchronous communication? An experimental study. User Model. User-Adap. Inter. 16(2), 87–127 (2006)
Ogata K.: Modern Control Engineering. 5th edn. Prentice Hall, Upper Saddle River, NJ (2009)
Ohene-Djan, J.: Ownership transfer via personalisation as a value-adding strategy for web-based education. In: Workshop on Adaptive Systems for Web-Based Education at AH2002, pp. 27–41. Málaga, Spain (2002)
Oliver, N., Horvitz, E.: A comparison of HMMs and dynamic bayesian networks for recognizing office activities. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 199–209. Springer, Berlin (2005)
O’Malley, C.E., Draper, S.W., Riley, M.S.: Constructive interaction: A method for studying Human–Computer-human interaction. In: 1st International Conference on Human–Computer Interaction, pp. 269–274. Honolulu, HI (1984)
Oppermann R.: Adaptively supported adaptability. Int. J. Hum. Comput. Stud. 40(3), 455–472 (1994)
Oppermann R.: Introduction. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 1–13. Lawrence Erlbaum Associates, Hillsdale (1995)
Ortigosa, A., Carro, R. M.: The continuous empirical evaluation approach: Evaluating adaptive web-based courses. In: 9th International Conference on User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 163–167. Springer, Berlin (2003)
Paek T., Chickering D.M.: Improving command and control speech recognition on mobile devices: using predictive user models for language modelling. User Model. User-Adap. Inter. 17(1-2), 93–117 (2007)
Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 9–24. Sonthofen, Germany (2001)
Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 438–442 (2005)
Person, N.K., Graesser, A.C.: Tutoring research group human or computer? AutoTutor in a bystander Turing test. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France. LNCS, vol. 2363, pp. 821–830. Springer, Berlin (2002)
Petrelli D., Not E.: User-centred design of flexible hypermedia for a mobile guide: reflections on the HyperAudio experience. User Model. User-Adap. Inter. 15(3-4), 303–338 (2005)
Pohl, W.: LaboUr—Machine learning for user modeling. In: 7th International Conference on Human–Computer Interaction, pp. 27–30. Elsevier, Amsterdam (1997)
Pohl W.: Logic-based representation and reasoning for user modeling shell systems. User Model. User-Adap. Inter. 9(3), 217–282 (1999)
Popescu, E.: Evaluating the impact of adaptation to learning styles in a web-based educational system. In: 8th International Conference on Web-Based Learning, Aachen, Germany. LNCS, vol. 5686, pp. 343–352. Springer, Berlin (2009)
Porayska-Pomsta K., Mavrikis M., Pain H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User-Adap. Inter. 18(1-2), 125–173 (2008)
Preece J., Rogers Y., Sharp H., Benyon D.: Human–Computer Interaction. Addison-Wesley, Reading (1994)
Robson C.: Experiment, Design, and Statistics in Psychology. 3rd edn. Penguin, London (1994)
Santos, O., Boticario, J.: Guiding learners in learning management systems through recommendations. In: 4th European Conference on Technology Enhanced Learning, Nice, France. LNCS, vol. 5794, pp. 596–601. Springer, Berlin (2009)
Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start collaborative filtering. In: 25th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, pp. 253–260. ACM, New York (2002)
Schmidt, D., Zukerman, I., Albrecht, D.: Assessing the impact of measurement uncertainty on user models in spatial domains. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 210–222. Springer, Berlin (2009)
Scriven M.: Produce evaluation. In: Smith, N.L. (ed.) New Techniques for Evaluation, pp. 121–126. Sage, Beverly Hills (1981)
Scriven M.: Beyond formative and summative evaluation. In: McLaughlin, G.W., Phillips, D.C. (eds) Evaluation and Education: At Quarter Century, pp. 19–64. University of Chicago Press, Chicago (1991)
Scriven M.: Types of evaluation and types of evaluator. Eval. Pract. 17(2), 151–162 (1996)
Serna A., Pigot H., Rialle V.: Modeling the progression of Alzheimer’s disease for cognitive assistance in smart homes. User Model. User-Adap. Inter. 17(4), 415–438 (2007)
Shneiderman B.: Designing the User Interface: Strategies for Effective Human–Computer Interaction. Addison Wesley, Reading (1998)
Sixsmith A.J.: An evaluation of an intelligent home monitoring system. J. Telemed. Telecare 6(2), 63–72 (2000)
Spada, D., Sánchez-Montañés, M., Paredes, P., Carro, R.: Towards inferring sequential-global dimension of learning styles from mouse movement patterns. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 337–340. Springer, Berlin (2008)
Stary, C., Totter, A.: How to integrate concepts of the design and the evaluation of adaptable and adaptive user interfaces. In: 3rd ERCIM Workshop on User Interfaces for All, pp. 68–75. Obernai, France (1997)
Stamou S., Ntoulas A.: Search personalization through query and page topical analysis. User Model. User-Adap. Inter. 19(1), 5–33 (2009)
Stock O., Zancanaro M.: PEACH—Intelligent Interfaces for Museum Visits. Springer, Berlin (2007)
Stock O., Zancanaro M., Busetta P., Callaway C., Krüger A., Kruppa M., Kuflik T. et al.: Adaptive, intelligent presentation of information for the museum visitor in PEACH. User Model. User-Adap. Inter. 17(3), 257–304 (2007)
Suebnukarn S., Haddawy P.: Modeling individual and collaborative problem-solving in medical problem-based learning. User Model. User-Adap. Inter. 16(3-4), 211–248 (2006)
Tarpin-Bernard, F., Marfisi-Schottman, I., Habieb-Mammar, H.: AnAmeter: The first steps to evaluating adaptation. In: 6th Workshop on User-Centred Design and Evaluation of Adaptive Systems at UMAP2009, pp. 11–20. CEUR, Trento, Italy (2009)
Tintarev, N., Masthoff, J.: Effective explanations of recommendations: User-centered design. In: ACM conference on Recommender systems, pp. 153–156. ACM, Minneapolis, MN (2007)
Tintarev, N., Masthoff, J.: The effectiveness of personalized movie explanations: An experiment using commercial meta-data. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 204–213. Springer, Berlin (2008)
Tintarev, N., Masthoff, J.: Evaluating recommender explanations: problems experienced and lessons learned for the evaluation of adaptive systems. In: 6th Workshop on User Centered Design and Evaluation at UMAP09, pp. 54–63. CEUR, Trento, Italy (2009)
Tobar, C.M.: Yet another evaluation framework. In: Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 15–24. Johnstown, PA (2003)
Totterdell P., Boyle E.: The evaluation of adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 161–194. Academic Press, London (1990)
Totterdell P., Rautenbach P.: Adaptation as a problem of design. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 61–84. Academic Press, London (1990)
Totterdell P., Rautenbach P., Wilkinson A., Anderson S.: Adaptive interface techniques. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 131–160. Academic Press, London (1990)
Trewin, S.: Configuration agents, control and privacy. In: ACM Conference on Universal Usability, pp. 9–16. ACM, Arlington, VA (2000)
Turing A.: Computing machinery and intelligence. Mind 59, 433–460 (1950)
van Barneveld, J., van Setten, M.: Involving users in the design of user interfaces for TV recommender systems. In: 3rd Workshop on Personalization in Future TV at UM03, Johnstown, PA (2003)
van den Haak M.J., de Jong M.D.T., Schellens P.J.: Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 22(5), 339–351 (2003)
van den Haak M.J., de Jong M.D.T., Schellens P.J.: Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Interact. Comput. 16(6), 1153–1170 (2004)
van Velsen L., van der Geest T., Klaassen R., Steehouder M.: User-centered evaluation of adaptive and adaptable systems: a literature review. Knowl. Eng. Rev. 23(3), 261–281 (2008)
VanLehn, K., Niu, Z., Siler, S., Gertner, A.S.: Student modeling from conventional test data: a Bayesian approach without priors. In: 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1452, pp. 434–443. Springer, Berlin (1998)
Walker E., Rummel N., Koedinger K.R.: CTRL: A research framework for providing adaptive collaborative learning support. User Model. User-Adap. Inter. 19(5), 387–431 (2009)
Wang, Y., Chen, Z., Kobsa, A.: A collection and systematization of international privacy laws, with special consideration of internationally operating personalized websites. http://www.ics.uci.edu/~kobsa/privacy (2006)
Weber, G., Specht, M.: User modeling and adaptive navigation support in WWW-based tutoring systems. In: 6th International Conference on User Modeling, Chia Laguna, Italy, pp. 289–300. Springer, Vienna (1997)
Weibelzahl, S.: Evaluation of adaptive systems. In: 8th International Conference on User Modeling. LNCS, vol. 2109, pp. 292–294. Springer, Berlin (2001)
Weibelzahl, S.: Evaluation of adaptive systems. PhD Thesis, University of Trier, Germany (2003)
Weibelzahl, S.: Problems and pitfalls in evaluating adaptive systems. In: 4th Workshop on the Evaluation of Adaptive Systems at UM’05, pp. 57–66. Edinburgh, UK (2005)
Weibelzahl, S., Weber, G.: Evaluating the inference mechanism of adaptive learning systems. In: 9th International Conference of User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 154–168. Springer, Berlin (2003)
Wharton C., Rieman J., Lewis C., Polson P.: The cognitive walkthrough method: A practitioner’s guide. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 105–141. John Wiley & Sons, New York (1994)
Wilson J., Rosenberg D.: Rapid prototyping for user interface design. In: Helander, M. (ed.) Handbook of Human–Computer Interaction, pp. 859–875. Elsevier, Amsterdam (1988)
Winter, S., Wagner, S., Deissenboeck, F.: A comprehensive model of usability. In: Engineering Interactive Systems Conference. LNCS, vol. 4940, pp. 106–122. Springer, Berlin (2008)
Witten I.A., Frank E.: Data Mining: Practical Machine Learning Tools and Techniques. 2nd edn. Morgan Kaufmann, Amsterdam (2005)
Yang, D., Huo, H.: Assessment on the adaptivity of adaptive systems. In: International Conference on Management of e-Commerce and e-Government, pp. 437-440. IEEE, Nanchang, China (2008)
Yudelson M., Medvedeva O., Crowley R.: A multifactor approach to student model evaluation. User Model. User-Adap. Inter. 18(4), 349–382 (2008)
Zancanaro, M., Kuflik, T., Boger, Z., Goren-Bar, D., Goldwasser, D.: Analyzing museum visitors’ behavior patterns. In: 11th International Conference on User Modeling, Corfu, Greece. LNCS, vol. 4511, pp. 238–246. Springer, Berlin (2007)
Zaslow, J.: If TiVo thinks you are gay, here’s how to set it straight. Wall Street J. A 1 (Nov 26, 2002)
Zhang, T., Rau, P., Salvendy, G.: Developing instrument for handset usability evaluation: A survey study. In: 12th International Conference on Human–Computer Interaction, Beijing, China. LNCS, vol. 4550, pp. 662–671. Springer, Berlin (2007)
Ziegler J., Bullinger J.H.: Formal models and techniques in Human–Computer interaction. In: Shackel, B., Richardson, S.J. (eds) Human Factors for Informatics Usability, pp. 183–206. Cambridge University Press, Cambridge (1991)
Ziegler, C.-N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: 14 th International World Wide Web Conference, pp. 22–32. ACM, Chiba, Japan (2005)
Zimmermann A., Lorenz A.: LISTEN: A user-adaptive audio-augmented museum guide. User Model. User-Adap. Inter. 18(5), 389–416 (2008)
Zimmermann A., Specht M., Lorenz A.: Personalization and context management. User Model. User-Adap. Inter. 15(3-4), 275–302 (2005)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Paramythis, A., Weibelzahl, S. & Masthoff, J. Layered evaluation of interactive adaptive systems: framework and formative methods. User Model User-Adap Inter 20, 383–453 (2010). https://doi.org/10.1007/s11257-010-9082-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11257-010-9082-4