Keywords

1 Introduction

Nowadays, emotion plays a fundamental role in consumer product design. Thus, design process must include tools to evaluate emotions in interactions with objects in the world. Normally, researchers usually measure human emotion by subjective evaluation and objective usability evaluation, like questionnaires [2], Self-Assessment-Mannequin (SAM) [3], Emotional Engagement Scale (EES) [4], Cognitive Engagement Scale (CES) [5], System Usability Scale (SUS) [6], Lickert Scale [7], Kansei engineering and other design method, like user observation [8], interview [9], focus groups [10], cultural probes [11], etc. However, sometimes these kinds of method fails to reflect the real emotions of the user and even cause erroneous interpretations. What the customer says is really what do they feel? So, it is necessary for researchers to do some more studies on this field.

To be mentioned, some scholars have recently started using physiological parameter evaluate the emotions of human interaction with products, such as: Oliveira et al. [12] who evaluated the subjective emotion (valence, arousal and dominance) and heart rate responses, when the participants interact with two immersive Virtual Reality (VR) environment; Trindade et al. [13] who used a face-reading tool to measure the emotional participants reactions, when they play a digital game. They studied the sensitivity of the tool to measure the emotional reactions at the different moments of the game, and the relation with the emotion reactions and the usability problems of the game.

Some studies have shown that EEG, infrared thermography, eye tracking, the face reading and some other technology measurements can help reduce statistical errors in data and improve user perception in their interactions with products. Slobounov et al. [14] used EEG to exams the effect of fully immersive 3Dstereoscopic presentations and less immersive 2D VR environments on brain functions and behavioral outcomes. The study shows that using EEG may be a promising approach for performance enhancement and potential applications in clinical/rehabilitation settings; Guo et al. [15] designed Thirty-two 3D prototypes of LED desk lamp to simulate an aesthetic appreciation flow. The study aims to integrate eye-tracking metrics and EEG measurements to distinguish and quantify the visual aesthetics of a product. The quantification method can help designers measure the visual aesthetics of their products and reduce errors in design; Soares et al. [16] use a subjective evaluation scale and the infrared thermography to measure what the user felt when he was handling a product control device; Barros et al. [7] conducted a usability evaluation of how users manually handle PET bottles for soft drinks by using Lickert scale, eye tracking and EEG.

This paper introduces a systematic review on emotion including Kansei Engineering and biofeedback, such as neuroergonomics and neurodesign tools (Eletroencephalography-EEG and Infrared Thermography).

2 Method

2.1 Systematic Review and Data Mining

Nowadays, a large number of scientific literature and various research results have emerged. The growth rate of journal literature and various monographs has far exceeded the scope of people’s reading ability. In order to save reading time and obtain the most relevant information it is possible to carry out a systematic review.

Systematic review can provide a lot of information and knowledge for a certain field and specialty. Systematic review, as a relatively new comprehensive evaluation method of literature, has been widely used in many disciplines, especially the medical field in recent years.

Cook and Deborah (1997) [17] defined a systematic review as a secondary literature research method that aggregates information, critically evaluates the information, and synthesizes the results of the initial research from multiple perspectives. They point out three characteristics that an excellent systematic review should possess, namely: Precise, explicit definition and statement of the research question to be treated; Reproducible search strategy (for this reason, the systematic review article should include the database, terms, and restrictions on the initial study year, language, etc.), emphasizing the repeatability of the research; use pre-set initial research inclusion and exclusion criteria.

This data mining process uses a visual method to conduct systematic review based on the concept of data mining [18], the data was subdivided into three main moments: problem identification, preprocessing and transformation. In this context, explain the Data Mining process by subdividing it into five steps (Fig. 1): (i) knowledge of the domain; (ii) preprocessing; (iii) pattern extraction; (iv) post-processing; (v) use of knowledge.

Fig. 1.
figure 1

Source: Blum, A.; Merino, E.A.D; Merino, G.S.A.D.

Data mining process: definition of graphic signals based on the organization.

Full size image

2.2 Research

  1. (i)

    Knowledge of domain. The first step resulted from the choice of databases according to their scope and availability description. And this review is based on human emotions when users interact with products. Thus, the research was applied to Web of science [19], Elsevier [20], ACM [21], SpringerLink [22] and Scopus [23], as they are multidisciplinary in scope and encompass the major areas of interest of the case study, especially Applied Social Sciences (where Design fits) and Health (focus on human and biofeedback).

  2. (ii)

    Preprocessing. Then, we proceeded to step preprocessing, when the period in which the data were to be retrieved, the type of material to be searched, the target areas of the research and the establishment of the descriptors were defined. As data retrieval period, we searched the publications made from the year 2000.

As a type of material to be searched, the search for complete articles was determined within the major areas of interest: Applied Social Sciences, Health and the like-these as main. Some bases, however, allowed to point out the areas of interest of the research more specifically and, according to the indicated ranges and the relationship with the intended focus, were selected as follows: Social Sciences, Social Technology and Arts Humanities at Web of science; Full-Text collection of the ACM; Full-Text collection of SpringerLink; Social Sciences & Humanities, Health Sciences and Life Sciences at Scopus.

It is the processing that data receives in order to be used in later steps. And on this step, the descriptors (keywords) was defined as Product design, EEG, Infrared Thermography, Kansei Engineer and Emotion (Fig. 2).

Fig. 2.
figure 2

Source: the authors.

Descriptors defined in step (ii) preprocessing.

Full size image

Subsequently, they were organized, in a combined manner, in step (iii) pattern extraction, when the Boolean operators “AND” were used. The terms, in English, were combined to promote the collection of data that covered from macro to micro approach, as the research interests.

  1. (iii)

    Pattern extraction. The use of Boolean operators allowed an organized approach in the following fields: (A) Use EEG and Infrared Thermography to test product design; (B) Use EEG, Infrared Thermography and Kansei Engineer method to test product design; (C) Use EEG, Infrared Thermography and Kansei Engineer method to measure user emotion by testing product design; (D) Use Infrared Thermography and Kansei Engineer method to test product design; (E) Use EEG to measure user emotion by testing product design; (F) EEG, Infrared Thermography and Kansei Engineer methodology; (G) Use EEG, Infrared Thermography and Kansei Engineer method to measure user emotion.

For better visualization to help the search process, these fields were organized using symbols and colors that acted as a query scheme throughout the search. Figure 3 shows the representation of fields A, B and C; D and E; F and G.

Fig. 3.
figure 3

Source: the authors.

Combination of descriptors in step (iii) pattern extraction.

Full size image

Fields A through C brought together terms focused on “using EEG and Infrared Thermography test product” and were structured as follows: A = [(product design) AND (EEG) AND (Infrared Thermography)]; B = [(A) AND (Kansei Engineer)]; C = [(B) AND (emotion)]. Likewise, the fields D and E - which brought together terms focusing on “using Infrared Thermography and Kansei engineer test product” - followed the same logic: D = [(product design) AND (Infrared Thermography) AND (Kansei Engineer)]; E = [(D) AND (emotion)];

Similar to the organizations in the previous fields, F to G have put together terms that focus on “EEG, Infrared Thermography and Kansei Engineer”: F = [(EEG) AND (Infrared Thermography) AND (Kansei Engineer)]; G = [(F) AND (emotion)].

The combination and division by fields allowed the extraction of patterns that resulted in a quantitative survey of articles. Table 1 shows the results of the survey conducted between January 10–15th, 2020. The highlighted numbers correspond to the articles that were analyzed in more detail in the subsequent step.

Table 1. Table indicating the number of articles raised in the bases from the application of step (iii) pattern extraction.
Full size table
  1. (iv)

    Post-processing. Considering Data Mining by grouping, post-processing step further analyzed the numerical results obtained. The results over one hundred were again mined from title and keyword analysis.

As can be seen from Table 1, in the field A, there only one paper-Comparing Thermographic, EEG, and Subjective Measures of Affective Experience During Simulated Product Interactions [24]-from Web of science related to the field A; Two papers from Scopus, one is same as the one from Web of science, another is “Comparison of thermographic, EEG and subjective measures of affective experience of designed stimuli [25]”.

The field B and C have same papers: (i) One is from the 23rd annual meeting of the Japan neuroscience society and the 10th annual meeting of the Japanese neural network society [26]. (ii) And one paper - Proposal for Indices to Assess Attractiveness on Initial Use of Mobile Phones [27]-is form SpringerLink.

The field D and E have one same papers from ACM: It is “A system for embodied social active listening to sound and music content [28].” from ACM; To be mentioned, the field D has two more related papers from ACM than field E from ACM: (i) one is “Kansei: a testbed for sensing at scale [29]”; (ii) another one is “Implementing an autonomic architecture for fault-tolerance in a wireless sensor network testbed for at-scale experimentation [30]”.

The field D and E have three same papers from SpringerLink: One is “Using Digital Thermography to Analyse the Product User’s Affective Experience of a Product [31]”; Another one is “Application of Digital Infrared Thermography for Emotional Evaluation: A Study of the Gestural Interface Applied to 3D Modeling Software [16]”; And one paper same as the one from (iii) of field B and C.

The field F has 234 papers from ACM and three from SpringerLink: (i) One related paper is “Dynamic analysis of dorsal thermal images [32]”; (ii) And another one is “Analysis of Product Use by Means of Eye Tracking and EEG: A Study of Neuroergonomics [7]”; (iii) The last one is same as the one from (iii) of field B and C.

The field G has 140 papers from ACM and two papers from SpringerLink: one is same as the one from (ii) of field B and C; and another one is same as the one from (ii) of field F.

Subsequently, the articles were appreciated through the abstract and framed in “related topics”. Of these, those that dealt with subjects directly related to the research theme were classified as “direct relation” (Table 2).

Table 2. Table indicating the number of articles on the subject of interest from the application of step (iv) post processing.
Full size table
  1. (v)

    Use of knowledge. As a result of the applied process, the papers listed under “Related topics”, two were available on more than one basis. The “related topics” totaled 30 papers. Of these, seven were directly related to the research theme, that is, studies involving elements of product design, measure emotion by using biofeedback or EEG or Infrared Thermography or Kansei engineer. They are: Jenkins, Brown and Rutterford (2009) [24]; Lennart (2010) [33]; Jaichandar, Elara and Edgar (2012) [34]; Yamagishi et al. (2011) [27].

3 Findings

This study have found 740 related papers from thousands of articles, and selected 151 related topic papers by title and keywords. Further read the content and output of these 151 papers, and finally select 30 papers related to “Using biofeedback (EEG And Infrared Thermography) to evaluate emotion And user Perception acquired by Kansei Engineering”.

From the data of Table 2, the number of papers related to the research topic is relatively small (30 papers), which indicates that there are relatively few studies focus on emotion recognition by using the biofeedback (EEG and Infrared Thermography) and Kansei Engineer method. From the content of the papers, the majority of emotion measurement is performed by using EEG or Infrared Thermography (16 papers. To be mentioned there are only two related papers using EEG and Infrared Thermography measure participants’ recognition or emotion. And seven papers mentioned other biofeedback measure method. There are five papers that use the combination of physiological data and subjective data measurement method; And any paper that uses Kansei engineer method and physiological data measurement to study user emotions.

Seven Directly Related Papers

  1. (i)

    “Comparing Thermographic, EEG, and Subjective Measures of Affective Experience During Simulated Product Interactions”. In this paper, Jenkins, Brown and Rutterford [24] using Affective Self Report (ASR), EEG and Infrared Thermography measure cognitive work and affective state of Sixteen male volunteers (mean age = 21.75 years)’ cognitive.

  2. (ii)

    “Proposal for Indices to Assess Attractiveness on Initial Use of Mobile Phones”. In this study, Yamagishi et al. [27] measured physiological indices of attractiveness during participants initial use of a mobile, including measurements of the automatic nervous system, nasal skin temperature, pupil diameter, EEG, blinking and electrocardiography. This study measured Nasal Skin Temperature and Pupil Diameter of Eleven undergraduate and graduate participants (six men and five women, mean age = 22.1 years, SD = 1.30); tested Ten undergraduate and graduate participants (eight men and two women, mean age = 21.7 years, SD = 0.67) participated in the EEG experiment; and Ten undergraduate and graduate participants (10 men, mean age = 21.2 years, SD = 0.63) participated in the experiment to measure blinks and ECG.

  3. (iii)

    “Using Digital Thermography to Analyse the Product User’s Affective Experience of a Product”. This study conducted a usability evaluation of users during manual handling of soda PET packaging by comparing the user-reported experience and the actual experience felt measured through usability analysis techniques and thermography. Barros et al. [31] proved that thermography has proven to be effective to measure users’ satisfaction (felt experience) in handling consumer products. There are two field studies: Field Study I, 12 participants (8 female and 4 male) and Field Study II, 11 volunteers were selected (7 female and 4 male).

  4. (iv)

    “Application of Digital Infrared Thermography for Emotional Evaluation: A Study of the Gestural Interface”. In this paper, Soares, Vitorino and Marçal [16] studied the application of infrared digital thermography as a tool to evaluate the level of emotional stress during the use of a computer system. They using a subjective evaluation scale and the infrared thermography to measure what the user felt when he was handling a product control device. In the usability evaluation, a test was developed with a sequence of tasks performed by 12 volunteers.

  5. (v)

    “Analysis of Product Use by Means of Eye Tracking and EEG: A Study of Neuroergonomics”. Barros et al. [7] measured user satisfaction with soft drinks PET packaging by using Lickert scale assess, eye tracking and EEG. This study comprising 12 participants for the usability study who are from different age groups and are higher education students.

  6. (vi)

    “Wii remote vs. controller: electroencephalographic measurement of affective gameplay interaction”. In this paper, Lennart [33] studied the influence of interaction modes (Playstation 2 game controller vs. Wii remote and Nunchuk) on subjective experience and brain activity measured with Electroencephalographic measures and survey measures-a game experience questionnaire (GEQ). This study measured Thirty-six Swedish undergraduate university students and employees participated in this experiment. Their age ranged between 18 and 41, having an average (M) age of 24 (Standard Deviation [SD] = 4.9).

  7. (vii)

    “Investigation of facial infrared thermography during interaction with therapeutic pet robot during cognitive training: a quantitative approach”. Jaichandar et al. [34] using thermography validate the functional process involved in temperature and correlation between cognitive task. The participants were mainly students from the bioengineering option (Singapore Polytechnic), with a total of 11 students of ages from 18 to 21 years old with an average of 19.6 years old and a standard deviation of 1.2.

From the information above, we can see clearly that most of the experiments are focus on electronic products: three different models of mobile phones, gesture computer system, Wii mote, controller, and pet robots. The participants of the experiments were all young people with average ages between 19.6 to 24.

4 Conclusion

Purpose of this research is searching the related references on EEG, Infrared Thermography, Kansei Engineering and emotion. To achieve this aim, this study selected papers from five databases through a visual method to conduct systematic review based on the concept of data mining. Searching for the combination of five keywords “product design, EEG, Infrared Thermography, Kansei Engineer and emotion”.

Through the selection and reading of related topic papers, it is found that currently there are some research papers on biofeedback measurement of user emotions; but there are fewer articles combining physiological data measurement and psychological data measurement methods; Moreover, there is no research use biofeedback and Kansei engineer methods to study user emotions. This also reveals that this field is still in the preliminary research stage and is worth further research and discussion in the future.

There are some limitation of this research, because the database is too large, so this research set the following conditions: (i) The papers must be written in English; (ii) The search resources are limited to the selected five databases; (iii) The papers must be published from January 01, 2000 to January 15, 2020. It results the papers are relatively limited. Follow-up work, in future research, it is necessary to expand more databases, choose more language types, and extend the time. To ensure that the research is comprehensive and accurate.