Abstract
KANSEI Engineering (KE) [1] was created at Hiroshima University about 30 year ago and it is well known in the world at present as an ergonomic customer-oriented product development technology. It is a method for translating sensations and impressions into product parameters. The objective of KE is to study the relationship between product forms and KANSEI images. However, the KE method is based on the analysis of human subjective factors, customer’s psychological feelings and needs, which is transformed in product design parameters. The customer’s psychological feelings and needs are usually acquired by subjective tools. The questions which arises is if these subjective tools reflect the real customer needs. Nowadays, some scholars have recently started using biofeedback to evaluate the emotions of human interaction with products. Some studies have shown that EEG and Infrared Thermography measurements can help reduce subjective interpretation in data and improve user perception in their interactions with products. This systematic literature review aims to search the references on EEG, Infrared Thermography, Kansei Engineering and emotion. It will serve as a support for further researches to check if is possible to include biofeedback tools to contribute to subjective analyzes.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Consumer product
- Emotion
- Kansei Engineer
- EEG
- Infrared Thermography
- Product design
- Affective engineering
- Emotional design
1 Introduction
Nowadays, emotion plays a fundamental role in consumer product design. Thus, design process must include tools to evaluate emotions in interactions with objects in the world. Normally, researchers usually measure human emotion by subjective evaluation and objective usability evaluation, like questionnaires [2], Self-Assessment-Mannequin (SAM) [3], Emotional Engagement Scale (EES) [4], Cognitive Engagement Scale (CES) [5], System Usability Scale (SUS) [6], Lickert Scale [7], Kansei engineering and other design method, like user observation [8], interview [9], focus groups [10], cultural probes [11], etc. However, sometimes these kinds of method fails to reflect the real emotions of the user and even cause erroneous interpretations. What the customer says is really what do they feel? So, it is necessary for researchers to do some more studies on this field.
To be mentioned, some scholars have recently started using physiological parameter evaluate the emotions of human interaction with products, such as: Oliveira et al. [12] who evaluated the subjective emotion (valence, arousal and dominance) and heart rate responses, when the participants interact with two immersive Virtual Reality (VR) environment; Trindade et al. [13] who used a face-reading tool to measure the emotional participants reactions, when they play a digital game. They studied the sensitivity of the tool to measure the emotional reactions at the different moments of the game, and the relation with the emotion reactions and the usability problems of the game.
Some studies have shown that EEG, infrared thermography, eye tracking, the face reading and some other technology measurements can help reduce statistical errors in data and improve user perception in their interactions with products. Slobounov et al. [14] used EEG to exams the effect of fully immersive 3Dstereoscopic presentations and less immersive 2D VR environments on brain functions and behavioral outcomes. The study shows that using EEG may be a promising approach for performance enhancement and potential applications in clinical/rehabilitation settings; Guo et al. [15] designed Thirty-two 3D prototypes of LED desk lamp to simulate an aesthetic appreciation flow. The study aims to integrate eye-tracking metrics and EEG measurements to distinguish and quantify the visual aesthetics of a product. The quantification method can help designers measure the visual aesthetics of their products and reduce errors in design; Soares et al. [16] use a subjective evaluation scale and the infrared thermography to measure what the user felt when he was handling a product control device; Barros et al. [7] conducted a usability evaluation of how users manually handle PET bottles for soft drinks by using Lickert scale, eye tracking and EEG.
This paper introduces a systematic review on emotion including Kansei Engineering and biofeedback, such as neuroergonomics and neurodesign tools (Eletroencephalography-EEG and Infrared Thermography).
2 Method
2.1 Systematic Review and Data Mining
Nowadays, a large number of scientific literature and various research results have emerged. The growth rate of journal literature and various monographs has far exceeded the scope of people’s reading ability. In order to save reading time and obtain the most relevant information it is possible to carry out a systematic review.
Systematic review can provide a lot of information and knowledge for a certain field and specialty. Systematic review, as a relatively new comprehensive evaluation method of literature, has been widely used in many disciplines, especially the medical field in recent years.
Cook and Deborah (1997) [17] defined a systematic review as a secondary literature research method that aggregates information, critically evaluates the information, and synthesizes the results of the initial research from multiple perspectives. They point out three characteristics that an excellent systematic review should possess, namely: Precise, explicit definition and statement of the research question to be treated; Reproducible search strategy (for this reason, the systematic review article should include the database, terms, and restrictions on the initial study year, language, etc.), emphasizing the repeatability of the research; use pre-set initial research inclusion and exclusion criteria.
This data mining process uses a visual method to conduct systematic review based on the concept of data mining [18], the data was subdivided into three main moments: problem identification, preprocessing and transformation. In this context, explain the Data Mining process by subdividing it into five steps (Fig. 1): (i) knowledge of the domain; (ii) preprocessing; (iii) pattern extraction; (iv) post-processing; (v) use of knowledge.
2.2 Research
-
(i)
Knowledge of domain. The first step resulted from the choice of databases according to their scope and availability description. And this review is based on human emotions when users interact with products. Thus, the research was applied to Web of science [19], Elsevier [20], ACM [21], SpringerLink [22] and Scopus [23], as they are multidisciplinary in scope and encompass the major areas of interest of the case study, especially Applied Social Sciences (where Design fits) and Health (focus on human and biofeedback).
-
(ii)
Preprocessing. Then, we proceeded to step preprocessing, when the period in which the data were to be retrieved, the type of material to be searched, the target areas of the research and the establishment of the descriptors were defined. As data retrieval period, we searched the publications made from the year 2000.
As a type of material to be searched, the search for complete articles was determined within the major areas of interest: Applied Social Sciences, Health and the like-these as main. Some bases, however, allowed to point out the areas of interest of the research more specifically and, according to the indicated ranges and the relationship with the intended focus, were selected as follows: Social Sciences, Social Technology and Arts Humanities at Web of science; Full-Text collection of the ACM; Full-Text collection of SpringerLink; Social Sciences & Humanities, Health Sciences and Life Sciences at Scopus.
It is the processing that data receives in order to be used in later steps. And on this step, the descriptors (keywords) was defined as Product design, EEG, Infrared Thermography, Kansei Engineer and Emotion (Fig. 2).
Subsequently, they were organized, in a combined manner, in step (iii) pattern extraction, when the Boolean operators “AND” were used. The terms, in English, were combined to promote the collection of data that covered from macro to micro approach, as the research interests.
-
(iii)
Pattern extraction. The use of Boolean operators allowed an organized approach in the following fields: (A) Use EEG and Infrared Thermography to test product design; (B) Use EEG, Infrared Thermography and Kansei Engineer method to test product design; (C) Use EEG, Infrared Thermography and Kansei Engineer method to measure user emotion by testing product design; (D) Use Infrared Thermography and Kansei Engineer method to test product design; (E) Use EEG to measure user emotion by testing product design; (F) EEG, Infrared Thermography and Kansei Engineer methodology; (G) Use EEG, Infrared Thermography and Kansei Engineer method to measure user emotion.
For better visualization to help the search process, these fields were organized using symbols and colors that acted as a query scheme throughout the search. Figure 3 shows the representation of fields A, B and C; D and E; F and G.
Fields A through C brought together terms focused on “using EEG and Infrared Thermography test product” and were structured as follows: A = [(product design) AND (EEG) AND (Infrared Thermography)]; B = [(A) AND (Kansei Engineer)]; C = [(B) AND (emotion)]. Likewise, the fields D and E - which brought together terms focusing on “using Infrared Thermography and Kansei engineer test product” - followed the same logic: D = [(product design) AND (Infrared Thermography) AND (Kansei Engineer)]; E = [(D) AND (emotion)];
Similar to the organizations in the previous fields, F to G have put together terms that focus on “EEG, Infrared Thermography and Kansei Engineer”: F = [(EEG) AND (Infrared Thermography) AND (Kansei Engineer)]; G = [(F) AND (emotion)].
The combination and division by fields allowed the extraction of patterns that resulted in a quantitative survey of articles. Table 1 shows the results of the survey conducted between January 10–15th, 2020. The highlighted numbers correspond to the articles that were analyzed in more detail in the subsequent step.
-
(iv)
Post-processing. Considering Data Mining by grouping, post-processing step further analyzed the numerical results obtained. The results over one hundred were again mined from title and keyword analysis.
As can be seen from Table 1, in the field A, there only one paper-Comparing Thermographic, EEG, and Subjective Measures of Affective Experience During Simulated Product Interactions [24]-from Web of science related to the field A; Two papers from Scopus, one is same as the one from Web of science, another is “Comparison of thermographic, EEG and subjective measures of affective experience of designed stimuli [25]”.
The field B and C have same papers: (i) One is from the 23rd annual meeting of the Japan neuroscience society and the 10th annual meeting of the Japanese neural network society [26]. (ii) And one paper - Proposal for Indices to Assess Attractiveness on Initial Use of Mobile Phones [27]-is form SpringerLink.
The field D and E have one same papers from ACM: It is “A system for embodied social active listening to sound and music content [28].” from ACM; To be mentioned, the field D has two more related papers from ACM than field E from ACM: (i) one is “Kansei: a testbed for sensing at scale [29]”; (ii) another one is “Implementing an autonomic architecture for fault-tolerance in a wireless sensor network testbed for at-scale experimentation [30]”.
The field D and E have three same papers from SpringerLink: One is “Using Digital Thermography to Analyse the Product User’s Affective Experience of a Product [31]”; Another one is “Application of Digital Infrared Thermography for Emotional Evaluation: A Study of the Gestural Interface Applied to 3D Modeling Software [16]”; And one paper same as the one from (iii) of field B and C.
The field F has 234 papers from ACM and three from SpringerLink: (i) One related paper is “Dynamic analysis of dorsal thermal images [32]”; (ii) And another one is “Analysis of Product Use by Means of Eye Tracking and EEG: A Study of Neuroergonomics [7]”; (iii) The last one is same as the one from (iii) of field B and C.
The field G has 140 papers from ACM and two papers from SpringerLink: one is same as the one from (ii) of field B and C; and another one is same as the one from (ii) of field F.
Subsequently, the articles were appreciated through the abstract and framed in “related topics”. Of these, those that dealt with subjects directly related to the research theme were classified as “direct relation” (Table 2).
-
(v)
Use of knowledge. As a result of the applied process, the papers listed under “Related topics”, two were available on more than one basis. The “related topics” totaled 30 papers. Of these, seven were directly related to the research theme, that is, studies involving elements of product design, measure emotion by using biofeedback or EEG or Infrared Thermography or Kansei engineer. They are: Jenkins, Brown and Rutterford (2009) [24]; Lennart (2010) [33]; Jaichandar, Elara and Edgar (2012) [34]; Yamagishi et al. (2011) [27].
3 Findings
This study have found 740 related papers from thousands of articles, and selected 151 related topic papers by title and keywords. Further read the content and output of these 151 papers, and finally select 30 papers related to “Using biofeedback (EEG And Infrared Thermography) to evaluate emotion And user Perception acquired by Kansei Engineering”.
From the data of Table 2, the number of papers related to the research topic is relatively small (30 papers), which indicates that there are relatively few studies focus on emotion recognition by using the biofeedback (EEG and Infrared Thermography) and Kansei Engineer method. From the content of the papers, the majority of emotion measurement is performed by using EEG or Infrared Thermography (16 papers. To be mentioned there are only two related papers using EEG and Infrared Thermography measure participants’ recognition or emotion. And seven papers mentioned other biofeedback measure method. There are five papers that use the combination of physiological data and subjective data measurement method; And any paper that uses Kansei engineer method and physiological data measurement to study user emotions.
Seven Directly Related Papers
-
(i)
“Comparing Thermographic, EEG, and Subjective Measures of Affective Experience During Simulated Product Interactions”. In this paper, Jenkins, Brown and Rutterford [24] using Affective Self Report (ASR), EEG and Infrared Thermography measure cognitive work and affective state of Sixteen male volunteers (mean age = 21.75 years)’ cognitive.
-
(ii)
“Proposal for Indices to Assess Attractiveness on Initial Use of Mobile Phones”. In this study, Yamagishi et al. [27] measured physiological indices of attractiveness during participants initial use of a mobile, including measurements of the automatic nervous system, nasal skin temperature, pupil diameter, EEG, blinking and electrocardiography. This study measured Nasal Skin Temperature and Pupil Diameter of Eleven undergraduate and graduate participants (six men and five women, mean age = 22.1 years, SD = 1.30); tested Ten undergraduate and graduate participants (eight men and two women, mean age = 21.7 years, SD = 0.67) participated in the EEG experiment; and Ten undergraduate and graduate participants (10 men, mean age = 21.2 years, SD = 0.63) participated in the experiment to measure blinks and ECG.
-
(iii)
“Using Digital Thermography to Analyse the Product User’s Affective Experience of a Product”. This study conducted a usability evaluation of users during manual handling of soda PET packaging by comparing the user-reported experience and the actual experience felt measured through usability analysis techniques and thermography. Barros et al. [31] proved that thermography has proven to be effective to measure users’ satisfaction (felt experience) in handling consumer products. There are two field studies: Field Study I, 12 participants (8 female and 4 male) and Field Study II, 11 volunteers were selected (7 female and 4 male).
-
(iv)
“Application of Digital Infrared Thermography for Emotional Evaluation: A Study of the Gestural Interface”. In this paper, Soares, Vitorino and Marçal [16] studied the application of infrared digital thermography as a tool to evaluate the level of emotional stress during the use of a computer system. They using a subjective evaluation scale and the infrared thermography to measure what the user felt when he was handling a product control device. In the usability evaluation, a test was developed with a sequence of tasks performed by 12 volunteers.
-
(v)
“Analysis of Product Use by Means of Eye Tracking and EEG: A Study of Neuroergonomics”. Barros et al. [7] measured user satisfaction with soft drinks PET packaging by using Lickert scale assess, eye tracking and EEG. This study comprising 12 participants for the usability study who are from different age groups and are higher education students.
-
(vi)
“Wii remote vs. controller: electroencephalographic measurement of affective gameplay interaction”. In this paper, Lennart [33] studied the influence of interaction modes (Playstation 2 game controller vs. Wii remote and Nunchuk) on subjective experience and brain activity measured with Electroencephalographic measures and survey measures-a game experience questionnaire (GEQ). This study measured Thirty-six Swedish undergraduate university students and employees participated in this experiment. Their age ranged between 18 and 41, having an average (M) age of 24 (Standard Deviation [SD] = 4.9).
-
(vii)
“Investigation of facial infrared thermography during interaction with therapeutic pet robot during cognitive training: a quantitative approach”. Jaichandar et al. [34] using thermography validate the functional process involved in temperature and correlation between cognitive task. The participants were mainly students from the bioengineering option (Singapore Polytechnic), with a total of 11 students of ages from 18 to 21 years old with an average of 19.6 years old and a standard deviation of 1.2.
From the information above, we can see clearly that most of the experiments are focus on electronic products: three different models of mobile phones, gesture computer system, Wii mote, controller, and pet robots. The participants of the experiments were all young people with average ages between 19.6 to 24.
4 Conclusion
Purpose of this research is searching the related references on EEG, Infrared Thermography, Kansei Engineering and emotion. To achieve this aim, this study selected papers from five databases through a visual method to conduct systematic review based on the concept of data mining. Searching for the combination of five keywords “product design, EEG, Infrared Thermography, Kansei Engineer and emotion”.
Through the selection and reading of related topic papers, it is found that currently there are some research papers on biofeedback measurement of user emotions; but there are fewer articles combining physiological data measurement and psychological data measurement methods; Moreover, there is no research use biofeedback and Kansei engineer methods to study user emotions. This also reveals that this field is still in the preliminary research stage and is worth further research and discussion in the future.
There are some limitation of this research, because the database is too large, so this research set the following conditions: (i) The papers must be written in English; (ii) The search resources are limited to the selected five databases; (iii) The papers must be published from January 01, 2000 to January 15, 2020. It results the papers are relatively limited. Follow-up work, in future research, it is necessary to expand more databases, choose more language types, and extend the time. To ensure that the research is comprehensive and accurate.
References
Schütte, S.T.W., Eklund, J., Axelsson, J.R.C., et al.: Concepts, methods and tools in Kansei engineering. Theor. Issues Ergon. Sci. 5(3), 214–231 (2004)
Lietz, P.: Research into questionnaire design. Int. J. Market Res. 52(2), 249–272 (2010)
Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment Manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)
Karsaklian, E., Sorbello, C., Sorbello, A.: Mapping the pathway to emotional engagement a methodology to create the emotional engagement model. J. Acad. Bus. Econ. 17, 47–56 (2017)
Greene, B.A.: Measuring cognitive engagement with self-report scales: reflections from over 20 years of research. Educ. Psychol. 50(1), 14–30 (2015)
Barros, R.Q., Santos, G., Ribeiro, C., Torres, R., Barros, M.Q., Soares, M.M.: A usability study of a brain-computer interface apparatus: an ergonomic approach. In: Marcus, A. (ed.) DUXU 2015. LNCS, vol. 9186, pp. 224–236. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20886-2_22
Barros, R.Q., et al.: Analysis of product use by means of eye tracking and EEG: a study of neuroergonomics. In: Marcus, A. (ed.) DUXU 2016. LNCS, vol. 9747, pp. 539–548. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40355-7_51
Wimmer, K., Stiles, J.: The observational research handbook: understanding how consumers live with your products. J. Advert. Res. 41(1), 91–93 (2001)
Creusen, M., Hultink, E.J., Eling, K.: Choice of consumer research methods in the front end of new product development. Int. J. Market Res. 55(1), 81–104 (2013)
Bruseberg, A., McDonagh-Philp, D.: Focus groups to support the industrial/product designer: a review based on current literature and designers’ feedback. Appl. Ergon. 33(1), 27–38 (2002)
Gaver, W.W.: Cultural probes and the value of uncertainty. Interactions 11(5), 53–56 (2004)
Oliveira, T., Noriega, P., Rebelo, F., Heidrich, R.: Evaluation of the relationship between virtual environments and emotions. In: Rebelo, F., Soares, M. (eds.) AHFE 2017. AISC, vol. 588, pp. 71–82. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-60582-1_8
Trindade, Y., Rebelo, F., Noriega, P.: Potentialities of a face reading tool to a digital game evaluation and development: a preliminary study. In: Rebelo, F., Soares, M. (eds.) AHFE 2017. AISC, vol. 588, pp. 371–381. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-60582-1_37
Slobounov, S.M., Ray, W., Johnson, B., et al.: Modulation of cortical activity in 2D versus 3D virtual reality environments: an EEG study. Int. J. Psychophysiol. 95(3), 254–260 (2015)
Guo, F., et al.: Distinguishing and quantifying the visual aesthetics of a product: an integrated approach of eye-tracking and EEG. Int. J. Ind. Ergon. 71, 47–56 (2019)
Soares, M.M., Vitorino, D.F., Marçal, M.A.: Application of digital infrared thermography for emotional evaluation: a study of the gestural interface applied to 3D modeling software. In: Rebelo, F., Soares, M.M. (eds.) AHFE 2018. AISC, vol. 777, pp. 201–212. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-94706-8_23
Cook, D.J.: Systematic reviewes: synthesis of best evidence for clinical decisions. Ann. Intern. Med. 126(5), 376 (1997)
Blum, A., Merino, E.A.D., Merino, G.S.A.D.: Visual method for systematic review in design based on concepts of Data Mining. Eugenio Andrés Díaz Merino DAPesquisa 11(16), 124–139 (2016)
Web of Science (2020). www.isiknowledge.com. Accessed 16 Jan 2020
Elsevier (2020). https://www.sciencedirect.com. Accessed 16 Jan 2020
ACM (2020). https://dl.acm.org. Accessed 16 Jan 2020
LNCS (2020). http://www.springer.com/lncs. Accessed 16 Jan 2020
Scopus (2020). https://www.scopus.com. Accessed 16 Jan 2020
Jenkins, S., Brown, R., Rutterford, N.: Comparing thermographic, EEG, and subjective measures of affective experience during simulated product interactions. Int. J. Des. 3(2), 53–65 (2009)
Jenkins, S., Brown, R., Rutterford, N.: Comparison of thermographic, EEG and subjective measures of affective experience of designed stimuli. In: Proceedings from the 6th Conference on Design and Emotion (2008)
Abstract of the joint meetings of the 23rd annual meeting of the Japan neuroscience society and the 10th annual meeting of the Japanese neural network society, 4–6 September 2000, Yokohama, Japan. Plenary Lecture. Neurosci. Res. 38(Suppl. 1), pp. S1–S189 (2000)
Yamagishi, M., Jingu, H., Kasamatsu, K., Kiso, H., Fukuzumi, S.: Proposal for indices to assess attractiveness on initial use of mobile phones. In: Marcus, A. (ed.) DUXU 2011. LNCS, vol. 6769, pp. 696–705. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21675-6_79
Volpe, G., Camurri, A.: A system for embodied social active listening to sound and music content. J. Comput. Cult. Heritage 4(1), 1–23 (2011)
Ertin, E., Arora, A., Ramnath, R., et al.: Kansei: a testbed for sensing at scale. In: Proceedings of the 5th International Conference on Information Processing in Sensor Networks (IPSN 2006), pp. 399–406. ACM, New York (2006)
Sridharan, M., Bapat, S., Ramnath, R., et al.: Implementing an autonomic architecture for fault-tolerance in a wireless sensor network testbed for at-scale experimentation. In: Proceedings of the 2008 ACM Symposium on Applied Computing (SAC 2008), pp. 1670–1676. ACM, New York (2008)
Barros, R.Q., Soares, M.M., Maçal, M.A., et al.: Using digital thermography to analyse the product user’s affective experience of a product. In: Rebelo, F., Soares, M. (eds.) Advances in Ergonomics in Design. Advances in Intelligent Systems and Computing, vol. 485, pp. 97–107. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41983-1_10
Nozawa, A., Takei, Y.: Dynamic analysis of dorsal thermal images. Artif. Life Rob. 16(2), 147–151 (2011)
Nacke, L.E.: Wiimote vs. controller: electroencephalographic measurement of affective gameplay interaction. In: Proceedings of the International Academic Conference on the Future of Game Design and Technology (Futureplay 2010), pp. 159–166. ACM, New York (2010)
Jaichandar, K.S., Elara, M.R., García, E.A.M.: Investigation of facial infrared thermography during interaction with therapeutic pet robot during cognitive training: a quantitative approach. In: Proceedings of the 6th International Conference on Rehabilitation Engineering & Assistive Technology (i-CREATe 2012), Article 28, pp. 1–4. Singapore Therapeutic, Assistive & Rehabilitative Technologies (START) Centre, Midview City (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zeng, J., Soares, M.M., He, R. (2020). Systematic Review on Using Biofeedback (EEG and Infrared Thermography) to Evaluate Emotion and User Perception Acquired by Kansei Engineering. In: Marcus, A., Rosenzweig, E. (eds) Design, User Experience, and Usability. Interaction Design. HCII 2020. Lecture Notes in Computer Science(), vol 12200. Springer, Cham. https://doi.org/10.1007/978-3-030-49713-2_40
Download citation
DOI: https://doi.org/10.1007/978-3-030-49713-2_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49712-5
Online ISBN: 978-3-030-49713-2
eBook Packages: Computer ScienceComputer Science (R0)